
Calhoun: The NPS Institutional Archive 
DSpace Repository 


Theses and Dissertations 


1. Thesis and Dissertation Collection, all items 


1995-12 

An analysis of cost overruns in the 
development of the Navy's A-12 Avenger 
aircraft [electronic resource] 

McKsymick, Eric M.; McKsymick, Eric M. 

Monterey, California. Naval Postgraduate School 


http://hdl.handle.net/10945/7472 


This publication is a work of the U.S. Government as defined in Title 17, United 
States Code, Section 101. Copyright protection is not available for this work in the 
United States. 


Downloaded from NPS Archive: Calhoun 



DUDLEY 

KNOX 

LIBRARY 


http ://w w w. nps.edu/lEbrary 


Calhoun is the Naval Postgraduate Schools public access digital repository for 
research materials and institutional publications created by tire NPS community, 
Calhoun is named for Professor of Mathematics Guy K. Calhoun, NPS's first 
appointed — and published — scholar^ author. 

Dudley Knox Library / Naval Postgraduate School 
411 Dyer Road / 1 University Circle 
Monterey, California USA 93943 


NAVAL POSTGRADUATE SCHOOL 
MONTEREY, CALIFORNIA 



To: Code 


_ 


From*- Thesis Processing. 


THESIS 


AN ANALYSIS OF COST OVERRUNS IN THE 
DEVELOPMENT OF THE NAVY’S A-12 
AVENGER AIRCRAFT 

by 

Eric M. McKsymick 
December 1995 

Thesis Advisor: Donald R. Eaton 


Thesis 

M236 


Approved for public release; distribution is unlimited. 








DUDLEY KNOX LIBRARY 
NAVAL POSTGRADUATE SCHOOL 
MONTEREY CA 93043-5101 






REPORT DOCUMENTATION PAGE 

Form Approved OMB No. 0704-0188 


Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188) 

Wash bet on DC 20503. 

1. AGENCY USE ONLY (Leave blank) 1 2. REPORT DATE 

December 1995 

3. REPORT TYPE AND DATES COVERED 
Master’s Thesis 

4. TITLE AND SUBTITLE AN ANALYSIS OF COST OVERRUNS IN THE 
DEVELOPMENT OF THE NAVY’S A-12 AVENGER AIRCRAFT 

5. FUNDING NUMBERS 

6. AUTHOR(S) ERIC M. MCKSYM1CK 

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

Naval Postgraduate School 

Monterev CA 93943-5000 

8. PERFORMING 

ORGANIZATION 

REPORT NUMBER 

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESSES) 

10. SPONSORING/MONITORING 
AGENCY REPORT NUMBER 

11 SUPPLEMENTARY NOTES The views expressed in this thesis are those of the author and do not reflect the 
official policy or position of the Department of Defense or the U.S. Government. 

12a. DISTRIBUTION/AVAILABILITY STATEMENT 

Approved for public release; distribution is unlimited. 

12b. DISTRIBUTION CODE 


13. abstract (maximum 200 words) This thesis examines the termination of the Navy’s A-12 
Program. Specifically, the research sought to answer the question: Were the A-12 
Program’s cost overruns exceptional when compared to other major acquisitions. Prior 
research indicates that most major programs excperience some degree of cost variance. To 
determine if the A-12's overruns were exceptional, the A-12 Program and 58 other contracts 
for developmental work are compared. The conclusion of the research is the A-12's overruns 
were exceptional. The cost overruns in the A-12 Program, at termination, exceeded 97 
percent of other programs examined. To complete the Program may have cost between $9 
and $11 billion. The required budget adjustment to complete the A-12 Program was greater 
than 91 percent of other programs. The research found no difference between cost variances 
of fixed-price contracts and cost-type contracts. The assertion that the use of a fixed-price 
contract contributed to the failure of the Program was not proven. There was also no 
statistical difference between the cost overruns of aircraft programs and other types of 
programs. The Government’s decision to terminate the A-12 Program for cost overruns is 
justified, based on the sample of programs examined. 


14 SUBJECT TERMS A-12, Cost Estimation, Cost Overruns 


19. SECURITY CLASSIFI¬ 
CATION OF ABSTRACT 
Unclassified 
































Approved for public release; distribution is unlimited 


AN ANALYSIS OF COST OVERRUNS IN THE DEVELOPMENT OF THE NAVY'S 
A-I2 AVENGER AIRCRAFT 


Eric M. McKsymick 
Captain, United States Army 
B.S., Illinois State University, 1985 


Submitted in partial fulfillment 
of the requirements for the degree of 


MASTER OF SCIENCE IN MANAGEMENT 


from the 








M ld(p 

t.K 



DUDLEY KNOX LIBRARY 

Basraas"* 


ABSTRACT 


This thesis examines the termination of the Navy's A-12 Program. Specifically, the 
research sought to answer the question: Were the A-12 Program's cost overruns 
exceptional when compared to other major acquisitions? Prior research indicates that 
most major programs experience some degree of cost variance. To determine if the 
A-12's overruns were exceptional, the A-12 Program and 58 other contracts for 
developmental work are compared. The conclusion of the research is the A-12's overruns 
were exceptional. The cost overruns in the A-12 Program, at termination, exceeded 97 
percent of other programs examined. To complete the Program may have cost between $9 
and $11 billion. The required budget adjustment to complete the A-12 Program was 
greater than 91 percent of other programs. The research found no difference between 
cost variances of fixed-price contracts and cost-type contracts. The assertion that the use 
of a fixed-price contract contributed to the failure of the program was not proven. There 
was also no statistical difference between the cost overruns of aircraft programs and other 
types of programs The Government's decision to terminate the A-12 Program for cost 


overruns is justified, based on the sample of programs examined. 
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f. INTRODUCTION 


A. OVERVIEW 

On 7 January 1991, Secretary ofDefense Richard Cheney announced the 
termination for default of the Full-Scale Development contract for the Navy's A-12 
Avenger program. The Secretary ofDefense cited cost overruns and schedule slips in his 
decision to cancel the A-12. The program was estimated to be $1 billion over cost and 
more than one year behind schedule. The Navy's internal investigation of the program. 

The Beach Report, cited numerous problems with the program [Ref. 2], 

In the same year Congressional Oversight Committees reviewed the Air Force's 
C-17 acquisition program. This program, like the A-12, was a fixed-price contract for 
development. In November of 1991 it was estimated to be at least $1 billion over cost and 
one year behind schedule. While the A-12 program was terminated, the C-17 continued 
into production. 

The Wall Street Journal called the litigation of the A-12 program the most costly 
case in history with legal costs alone running $30 million per year [Ref. 28, 27 July 1994] 
At stake is over $3 billion in taxpayer money, including $1.35 billion in overpayment of 
progress payments and a contractor claim of $1.9 billion. On 9 December 1994, U S. 
Claims Court Judge Robert Hodges vacated the Termination for Default and urged the 
Navy and the contractors to reach a settlement [Ref. 28, 12 December 1994], The Navy 
has decided to pursue the case in court and the case is expected to be heard in November 
of 1995. 

B. PURPOSE 

1. Thesis Objectives 

The intent of this research is to answer a straightforward question: Were the 
Navy's A-12 cost overruns exceptional when compared to other major acquisitions? The 
answer to this question is intended to be simple: yes or no. It is apparent from the history 
of major aircraft acquisitions that cost and schedule overruns alone do not account for the 
termination of programs. This research examines the cost variances within the A-12 





program, in comparison with other major acquisitions. Cost overruns can be attributable 
to many factors such as poor cost estimation, poor management or technical difficulty. 
The potential for costs overrunning the ceiling price was recognized within a year of the 
start of Full-Scale Development (FSD). According to an analyst at the Office of the 
Under Secretary of Defense for Acquisition and Technology (OUSD A&T), earlier 
accounting for these costs may have kept the program alive. [Ref. 4] 

2. Research Questions 

Primary Research Question: Were the cost overruns on the A-12 contract 
exceptional when compared to other major aircraft acquisitions and other major defense 
acquisitions? 

Subsidiary Research Questions: 

(1) What would the A-12 FSD program have cost if it continued to completion? 

(2) Was the Secretary of Defense justified in canceling the A-12 program? 

(3) Are there differences in cost overruns between fixed-price contracts and 
cost-type contracts? 

(4) Do differences exist in cost overruns of aircraft acquisitions and other major 
acquisitions? 

(5) What is the average cost overrun of a developmental contract? 

3. Expected Benefit 

This research is intended to determine if cost overruns in the A-12 program 
justified the termination for default when compared to other major aircraft acquisition 
programs. The primary benefit of this study is to provide insight into the effects of cost 
variances. This research provides the program manager with comparison data that can 
provide reference points for progress. 

4. Boundaries 

Only contracts for developmental programs are examined. No comparison is 
attempted to programs in earlier or later phases, and the reader is cautioned that 
techniques used in this study may not be applicable to other phases of the acquisition 
process. While there are studies that indicate many programs experience cost and 
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schedule variances, this study specifically examines the role of these variances in the 
decision to terminate the Navy's A-12 Avenger program. 

The intent of this research is to examine cost overruns in the A-12 program and 
provide a statistically valid finding on whether the A-12 program experienced 
extraordinary cost overruns. The methodology uses non-linear cost estimation models and 
non-parametric statistics. Recognizing that some readers of this study may not be familiar 
with these techniques many of the details of the computations are not included in the text. 
Except where required for background or clarity, the detailed methodology may be found 
in Appendix B. 

5. Limitations and Constraints 

The contracts included in this research had to meet several criteria. First, the 
contract had to be for the Engineering and Manufacturing Development (EMD) phase, 
previously designated the Full-Scale Development phase. (For continuity, the current 
designation of Phase II of the acquisition process will be used.) Second, the program had 
to meet the requirements to be an Acquisition Category ID (ACAT ID) program. That is, 
at least $200 million in Research, Development, Test and Evaluation (RDT&E) funds, in 
1980 constant dollars, had to have been committed to the program [Ref. 7], Third, the 
program must have achieved a significant degree of completion. The significant degree of 
completion was selected to be 40 percent. 

Only contracts from the Defense Acquisition Executive Summary (DAES) 
database were examined. Contracts for smaller programs were not examined. 

The determination of estimates at completion used a non-linear cost estimation 
model commonly called the Rayleigh Distribution. This model has the ability to provide 
estimates at completion that are independent of current budgets. There are many other 
models that provide estimates at completion that were not examined due to time 
constraints. No attempt is made to provide a general model for predicting costs at 
completion. 

The use of contractual information from the DAES database was contingent upon 
an agreement not to identify the contract or the contractor. Therefore, the contracts 
contained in Appendix A are listed by number for identification. Permission from OUSD 
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(A&T) is required prior to release of the database. The database used in this research was 
transferred to the Systems Management Department of the Naval Postgraduate School. 

6. Assumptions 

The approach used for this study was an examination of cost data from multiple 
contracts to determine if there were significant differences that would account for the 
termination of the A-12 based on cost variances. Cost information was obtained from the 
DAES database. Despite the judgmental selection of the contracts, an assumption is made 
that the contracts drawn from the database are representative of the population. 

The use of the Rayleigh Distribution for estimating the cost at completion of the 
A-l 2 program is based on prior research that indicates this model accurately represents 
the expenditures of a EMD program. It is assumed, based on past research, that the A-12 
program can be modeled using the Rayleigh Distribution. 

The effects of inflation were considered during the research. Inflation would have 
no impact on the cost variance between Budgeted Cost of Work Performed (BCWP) and 
the Actual Cost of Work Performed (ACWP). If these two figures were inflated to a base 
year amount the cost variance would remain the same. The second area inflation was 
considered is in determining adjustments to the Contract Budget Base. An assumption is 
made that contractors base the Budgeted Cost of Work Scheduled (BCWS) and the 
Estimate At Completion (EAC) on predictions of future price escalation. This assumption 
was confirmed by a prior program manager for Lockheed and by a cost analyst with the 
Cost Analysis Improvement Group (CAIG). No adjustments are made for inflation. 

C. METHODOLOGY 

1. A-12 Cost Estimation 

The estimates at completion for the A-12 were developed using a variant of the 
Rayleigh Distribution model. This model is defined in detail in Chapter H. The decision 
to use the Rayleigh model to estimate the completion cost is based on prior research 
conducted by Abemethy [Ref. 1], Elrod [Ref. 11], Lee [Ref. 18], Gallagher [Ref. 14] and 
Watkins [Ref. 29], 


2. The Contract Budget Base 

The use of cost variances alone does not provide a complete picture. A program 
that experiences significant cost variances will need an adjustment to its Contract Budget 
Base (CBB) if it is to continue. Ideally, there would be no cost variances within a 
program and no requirement to adjust the CBB. Both cost variances and CBB 
adjustments are examined in this research. 

3. Data Sources 

The data used for this study were drawn from 58 separate contracts for 
developmental work. Contract information was obtained from the OUSD (A&T). 
Supporting data were drawn from a variety of sources as listed in the references. 

4. Hypotheses 

a. Hypothesis 1: Cost overruns in the A-12 Program were no greater than 
the average cost overruns in the sample. This hypothesis was tested to determine if the 
A-12 cost variances and CBB adjustments were statistically different than other major 
programs. 

b. Hypothesis 2: p fp = p c . There are no differences in the cost variances of 
fixed-price contracts and cost-type contracts. 

c. Hypothesis 3: (i^ = n Mmp)e . There are no differences between cost 
variances of contracts for aircraft and other contracts in the sample. 

d. Hypothesis 4: p cbbf[) = p cbbc . There are no differences in the 
adjustments to the Contract Budget Base for fixed-price contracts and the adjustments to 
the Contract Budget Base for cost-type contracts. 

e. Hypothesis 5: p cbbacft = P cbb5a mpie ■ There are no differences in the 
adjustments to the Contract Budget Base of aircraft programs and other programs in the 
sample. 

Where p sample = the mean of the sample of ACAT ID contracts. p ae( , = the mean of 
aircraft contracts, = the mean of fixed-price contracts, and p c = the mean of cost 
contracts. Hypotheses four and five include the subscript CBB. Hypotheses four and five 
tested the contract budget base adjustments from the indicated samples. 

5. Procedure to Test the Hypotheses 

a. Hypothesis 1 was tested using the data from the A-12 reports and the 
data from the sample contracts. The data were measured at completion points from 0 to 




100 percent complete measured by dollars expended. The A-12 data will be compared 
based on percentile rankings. 

b. Hypotheses 2 thru 5 were tested using non-parametric statistics. The 
Mann-Whitney test was chosen for its ease of use. The data do not support an assumption 
of normality that would allow a t-test. Alpha was set at .05. 

6. Justification of Methodology 

Prior research confirmed that the Rayleigh Distribution patterns the expenditures in 
developmental programs. Elrod [Ref. 11] tested the ability of the Rayleigh Distribution to 
provide estimates at completion for aircraft programs and found that the Rayleigh 
Distribution can be used for this purpose. Lee [Ref. 18] developed a technique using the 
Rayleigh model to determine budgets for developmental programs. Gallagher [Ref. 14] 
developed a methodology that provided probabilities of ending cost estimates for 
developmental programs based on the. Rayleigh Distribution. 

The cost data from the A-12 program were used to determine a percentile value 
based on the distribution that best patterns the data. BestFit, a commercial statistical 
software package was used to determine the distributions based on Chi-squared values. 
The software evaluates the data versus 18 distributions to provide a range of possible 
distributions that model the data. Appendix B contains complete information on the 
modeling of the data. 

Testing non-normal distributed data requires tests not dependent on the parameters 
of the distribution. A non-parametric test is required for testing hypotheses on cost 
variance data. Cost variances for programs patterned a left skewed distribution. The 
addition of more data points did not normalize the distribution. This intuitively makes 
sense, if one considers there is little incentive for underruns in a program. 

In determining whether the cost variances of the A-12 were exceptional it is 
important to determine the CBB adjustments for other programs. Hypotheses four and 
five tested whether certain contracts or programs experienced greater adjustments to the 
budget than other programs. 
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D. LITERATURE REVIEW 

A review of the available literature revealed no directly related past effort. The 
most widely cited source in analyzing the failure of the A-12 program is the "Beach 
Report" [Ref 2], 

The Beach Report examined issues that contributed to the failure of the A-12 
program. Specifically, the Beach Report sought answers to three questions, "Did the 
Navy... have reason to anticipate substantial additional cost increase...at the time of the 
Major Aircraft Review? If not, why not? If so, were senior DON and DoD leaders 
sufficiently apprised in the course of the Review?" [Ref. 2: p. 2] The report concluded 
that sufficient cost performance data existed to indicate that the contract would have 
significant overruns and the Program Manager " . . .erred in judgment by failing to anticipate 
substantial additional cost increase beyond the ceiling of the FSD contract..." [Ref. 2: p. 
29], 

Previous studies have examined the affects of cost and schedule overruns on 
contract costs at completion. Dr. David S. Christensen, an Associate Professor of 
Accounting at The Air Force Institute of Technology, examined 64 contracts of various 
types and programs and the ability of a program to recover from cost overruns at various 
stages [Ref. 3], Using a linear regression model, Christensen tested the hypothesis that 
contracts were unlikely to recover from cost overruns. His results indicated that programs 
experiencing cost overruns, at a stage of completion between 10 to 70 percent, were 
highly unlikely to finish within the programmed cost baseline. Furthermore, he found that 
the type of contract used, the Service managing the program and the type of weapon being 
acquired had no significant bearing on the cost overrun incurred in the program. [Ref. 3] 

Christensen's work evolved from observations made by Mr. Gary Christie and Mr 
Wayne Abba, senior program analysts in the OUSD (A&T). Abba and Christie examined 
data extracted from the DAES database on 500 contracts and concluded there is very little 
chance of recovering from a cost overrun once a program is more than 15% complete. 
"Given a contract is more than 15 percent complete, the overrun at completion will not be 
less than the overrun to date, and the percent overrun at completion will be greater than 
the percent overrun to date." [Ref. 3] 
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E. REVIEW OF PRIOR RESEARCH 


Two master's theses from the Air Force Institute of Technology, Captain Scott R. 
Heise, A Review of Cost Performance Index Stability [Ref. 17], and Captain Brian D. 
Wilson, An Analysis of Contract Cost Overruns and Their Impacts [Ref 30], examined 
cost performance data from contracts. 

Captain Heise's research examined at what point the Cumulative Cost Performance 
Index (CPIcum) is considered to be stable. Data from the Office of the Under Secretary 
of Defense (Acquisition & Technology) indicate that the CPI is normally stable at the 50 
percent program completion point, and the cumulative CPI does not improve between 
program completion points of 15 percent to 85 percent. Captain Heise's hypothesis was 
that the CPI is stable when a program is greater than 50 percent complete. His definition 
of stability was that the CPI would not vary by more than plus or minus 10 percent. 

Using data extracted from the DAES database, Heise examined 155 contracts. 

The contracts included various acquisition phases and contract types. His conclusion was 
the cumulative CPI is stable at the 50 percent point and will stabilize as early as the 20 
percent completion point if the program maintains a stable baseline. Additionally, Captain 
Heise found that the cumulative CPI tends to worsen to some degree as the program 
continues, but not beyond the bounds of what is considered a stable CPI. His conclusions 
confirm previous work by Abba and Christie, that there is very little chance of recovery 
from a program cost overrun between the 15 percent and 85 percent completion points. 
[Ref. 17] 

Captain Brian D. Wilson's research investigated Christie's assertion that of 500 
contracts examined since 1977, the cost overrun at completion would be higher than the 
cost overrun to date. Captain Wilson examined 65 contracts, all of which had experienced 
cost overruns. His findings confirmed the claim. He also found that the cost overruns 
tend to increase as the program progresses. Captain Wilson's statistical research was 
performed at an 85 percent confidence level. [Ref. 30] 

Two master's theses from the Naval Postgraduate School were used for 
background information on the application of the Rayleigh Distribution to developmental 
programs. Abemethy [Ref. 1], investigated the applicability of the Rayleigh Distribution 



to programs and Elrod [Ref. 11], tested the ability of the Rayleigh Distribution to provide 
estimates at completion. 

Abernethy's work examined completed contracts to test the ability of the Rayleigh 
Distribution to model expenditures. The results showed that the expenditures of most 
contracts were modeled by the Rayleigh Distribution, but the predictive capability of the 
model was insufficient for use in obtaining accurate cost at completion estimates. [Ref. 1] 

Elrod tested the predictive capability of the Rayleigh Distribution using two 
methods to determine which method provided the best predictions of final cost. Her 
sample included aircraft programs that had been completed. Elrod's results showed that 
either of the two models could be used to predict final costs. The model used in this 
research approximates the Putnam model examined by Elrod. [Ref. 11] 

Two unpublished research papers were relied upon to provide an understanding of 
the application of the Rayleigh model.. The first paper, Determining a Budget Profile from 
a R & D Cost Estimate, by David A. Lee et. al [Ref. 18], provided verification that the 
Rayleigh model patterned the expenditures in a developmental program as well as a simple 
description of the model. This paper is available in the Defense Technical Information 
Center (DTIC) database. 

The second paper, Final-Cost Estimates for Research & Development Programs 
Conditioned on Realized Costs, by Mark A. Gallagher and David A. Lee [Ref. 14], 
provided a software tool that was used to develop estimates at completion for the A-12 
program and cumulative probabilities of occurrence. This paper is also available in the 
DTIC database, and was presented at the 1995 Military Operations Research Society 
Symposium. 

F. SUMMARY 

This Chapter presented the objectives of the research and the hypotheses to be 
tested Most programs experience cost overruns. Were the cost overruns in the A-12 
Program any different than other programs? 

The following chapters will provide the foundation to answer the question. 
Chapters II and III will present the general background information on the Acquisition 
Process and the A-12 Program. Chapter IV contains the data from the 58 contracts drawn 
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from the DAES database. Chapter V is the analysis of the data and the determination of 
where the A-12 Program was at versus other major acquisitions. Chapter VI reviews the 
research questions and provides the results of the tests of the hypotheses. 
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II. BACKGROUND 


A. OVERVIEW 


This chapter provides an overview of the acquisition process as it is currently 
structured, and a summary of the Cost/Schedule Control Systems Criteria. This chapter 
also introduces techniques for estimating costs at completion for an EMD program. 

The intent of this chapter is to provide the reader with sufficient background in the 
current Defense acquisition process, and cost estimating procedures, to better understand 
the events leading up to the termination of the A-12. The Acquisition Process information 
is drawn from OMB Circular A-109, and the DoD 5000 Series. 

B. THE ACQUISITION PROCESS 


1. The DoD 5000 Series 

The DoD 5000 series is a set of directives and instructions issued in 1991. The 
series consists of DoD Directive 5000.1, Defense Acquisition [Ref. 6]; DoD Instruction 
5000.2, Defense Acquisition Management Policies and Procedures [Ref. 7]; and DoD 
Manual 5000.2-M, Defense Acquisition Management Documents and Reports These 
replaced the 1987 versions of DoDD 5000.1 and DoDI 5000.2. 

All the military departments are subject to the guidance provided in the 5000 
series which provide a single acquisition system for all defense acquisition programs. The 
5000 series is implemented in a phased process with five major milestones, as shown in 
Figure 1. [Ref. 6] 



Figure 1. The Acquisition Process. [Ref. 6] 























2. The Mission Need 

The acquisition process begins with the identification of a deficiency in current 
tactics or equipment that can only be satisfied with the procurement of equipment. DoDD 
5000.1 requires that a new acquisition be initiated only after "fully examining alternative 
ways of satisfying identified military needs." The priority in examining alternatives as 
specified in DoDD 5000.1 is: [Ref. 6] 

1. Use or modification of an existing U.S. military system. 

2. Use or modification of an existing commercially developed or Allied 

system that fosters a non-developmental acquisition strategy. 

3. A cooperative research and development program with one or more 

Allied nations. 

4. A new Joint-Service development program. 

5. A new Service-unique development program. 

The increasing sophistication of threat air defense systems degraded the capability 
and survivability of the A-6. The concept exploration for the A-12 began at the end of the 
cold war and was focused on penetrating Soviet-type air defense systems using low 
observable (stealth) technology. The United States Air Force (USAF) systems that 
incorporated stealth technology were not suitable for carrier operations without significant 
structural modification that would result in a new aircraft from what was originally 
designed for USAF missions. 

The initiation of the A-12 program was based on projected USAF purchases of the 
aircraft in larger quantities than the Navy. Although this would not be a true "joint" 
program, the A-12 procurement was based on two Service's needs. 

The Services submit the Mission Need Statement (MNS) to a Joint Requirements 
Oversight Council (JROC). The JROC determines if the need can be satisfied by other 
means. The JROC determines the validity of the need and forwards the MNS to the 
OUSD (A&T) for approval or disapproval. 

The A-12 Program began in 1984 with approval of Milestone 0, approval for 
concept studies. The Navy contracts for Concept Exploration were awarded to the 
contractor teams of McDonnell Douglas/General Dynamics and Northrop/Grumman. 
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3. Phase 0 

Phase 0 is Concept Exploration and Definition. During this phase the priority is to 
develop and evaluate various alternatives that could satisfy requirements In Phase 0 
initial cost estimates are conducted to determine the affordability of the proposed systems. 
Trade-offs are considered between cost and performance. The contract for Phase 0 in the 
A-12 program was issued in 1984. Phase 0 of the acquisition process begins with 
Milestone 0, and approval of the Mission Need Statement (MNS). 

DoDI 5000.2 requires that during this phase plans must be made for competitive 
prototyping unless the OUSD (A&T) decides that it is not practical. Plans must also be 
made to allow for competitive development and production. 

4. Phase I 

Milestone I is the start of a new acquisition program for ACAT ID programs. 
ACAT ID programs are defined as programs that will expend more than $200 million in 
1980 constant dollars in RDT&E or the expenditure for procurement of more than $1 
billion in 1980 constant dollars. The decision authority to start a new program is the 
OUSD (A&T). The most promising designs from Phase 0 are carried forward into Phase 
I, the Demonstration and Validation Phase. 

The Defense Acquisition Board (DAB) reviews the program at each milestone to 
determine if it meets the requirements to proceed to the next milestone. If a new program 
is required, the parameters for the development of the system are established and 
approved. 

The threat is re-evaluated at Milestone I to ensure the requirements for the 
program still exist and have not changed. Total costs of the program are refined and 
affordability issues are considered. The intent of the Milestone decision is to ensure that a 
new program is not started when other means can satisfy the need. 

The A-12 Program entered Phase I in 1986, with the award of the Demonstration 
and Validation contract to McDonnell Douglas/General Dynamics and 
Northrop/Grumman; the same teams that had conducted the concept exploration studies. 
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Phase I continues to define the critical design characteristics and projected 
capabilities of the system. Projected costs of the system are further defined as the 
program proceeds through the Demonstration and Validation Phase. 

Risk is considered throughout the phases, but at this point the degree of risk of 
incorporating new technologies should be carefully considered. At the conclusion of 
Phase I the design to enter the Engineering and Manufacturing Development Phase is 
selected, and the design concepts should be relatively firm. 

5. Phase II 

Milestone II is the approval Milestone by the DAB to enter the Engineering and 
Manufacturing Development phase of a new system. Of crucial importance at the 
Milestone II review is the assessment that "the technologies and processes critical to 
success are attainable." The DAB is additionally charged with a "rigorous" assessment of 
the costs of the system. Prior to approval to enter EMD it should be demonstrated 
through testing that the system meets contract specification requirements, and operational 
requirements. 

The focus during EMD is on the producibility of the system. The A-12 contract for 
EMD was awarded to the team of McDonnell Douglas and General Dynamics in January 
1988, four years after initial concept studies had begun. The contractor team proposed an 
aggressive development schedule that would result in first flight thirty months after 
contract award. 

6. Phase HI 

Phase III is the production of the actual system. Milestone III is a critical 
milestone in a program's lifecycle. Milestone III is the Production Approval for the 
system. The objectives as defined in DoD 5000.2 are to: (1) determine if the results of 
Phase II warrant continuation, and (2) establish a Production Baseline containing refined 
program cost, schedule and performance objectives. [Ref. 7] Phase III is normally divided 
into two unofficial sub-phases: Low Rate Initial Production (LRIP) and Full Rate 
Production. The purpose of LRIP is to reduce the risk of producing the system by 
producing it in small quantities initially, then once the producibility is verified, to begin 
Full Rate Production. 
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7. Phase IV 

Phase IV usually completes the acquisition lifecycle. Phase IV is initiated by 
Milestone IV, Major Modification Approval. The purpose of the decision at Milestone IV 
is to determine if a major modification is required. In addition to major modifications. 
Phase IV includes the operations and support of the fielded equipment. 

C. COST AND SCHEDULE CONTROL SYSTEMS CRITERIA 

1. Overview 

Throughout the acquisition process the cost of the proposed system is of great 
importance. Cost must be considered at each milestone decision. All ACAT ID programs 
must incorporate C/S CSC into the management of the program. Cost and schedule 
control systems allow program managers to track expenditures of resources in the 
program. 

2. Requirements 

DoD 5000.2 requires that the contractor submit C/S CSC information on all 
significant contracts that are not firm fixed-price. The definition of a significant contract 
for the requirement is a contract with Research, Development, Test and Evaluation 
(RDT&E) values of $60 million or more, and procurement contracts valued at $250 
million or more, in fiscal year 1990 constant dollars. Contracts that are not determined to 
be significant must submit the Cost/Schedule Status Report (C/S SR). [Ref. 7] 

According to DoD 5000.2, the purposes of C/S CSC requirements are to: (1) 
provide uniform evaluation criteria to ensure contractor cost and schedule management 
systems are adequate, (2) to provide an adequate basis for responsible decision making by 
contractor management and DoD personnel, and (3) to bring to the attention of DoD 
contractors, and encourage them to accept and install, management control systems and 
procedures that are most effective in meeting requirements and controlling contract 
performance. [Ref. 7] 

Contractors report C/S CSC data monthly. DoD 5000.2 requires that 3 elements 
must be reported: (1) comparison of budgeted cost for work scheduled and the budgeted 
cost of work performed; (2) comparison of the budgeted cost of work performed and the 
actual cost of work performed; (3) variances resulting from the differences in comparisons 
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between budgeted cost of work schedule and performed, and variances between budgeted 
cost of work performed and actual cost of work performed. [Ref. 7] 

Of particular interest to program managers and DoD oversight personnel are the 
latest revised estimates (LRE) and estimates at completion (EAC) that are reported. The 
computation of these estimates may vary but there are four standard techniques that are 
explained later. 

3. Purpose of C/S CSC 

C/S CSC are a management tool for monitoring actual progress on a contract 
versus the baseline. C/S CSC begins with the formulation of a work breakdown structure 
(WBS) that identifies the tasks and sub-tasks that constitute the complete work package 
of the contract. The WBS identifies which particular tasks may be ahead or behind 
schedule and identifies areas where the reallocation of resources is required. The tasks 
and sub-tasks are provided parallel work package budgets that are traceable to the work 
being accomplished. 

There are several methods of allocating value. One method is when work package 
budgets are time phased; no dollars are credited to a work package not started. Once a 
work package is started, 50 percent of the designated budget is credited to the package, 
while the remaining 50 percent is credited at completion. An alternative means of 
allocating earned value is to assign 100 percent of the value of the work package only 
when it is completed. Some distortion will be experienced in the actual progress of the 
contract using either of these methods. 

The exact accounting system that a contractor must use to maintain C/S CSC data 
is not specified by the Government. A system that provides the required data does not 
need to be modified to meet contractual requirements. 

4. C/S CSC Elements 

Contractors submit the Cost Performance Report (CPR), which includes the C/S 
CSC data, to the program manager. For AC AT ID programs, program managers forward 
CPR information with the Defense Acquisition Executive Summary (DAES) Report to the 
OUSD (A&T). Within the OUSD (A&T) office a database is maintained of DAES CPR 
information. In this research, the following elements of the CPR are of importance: 



a. Budgeted Cost of Work Scheduled (BCWS) 

The BCWS is the dollar amount in current dollars budgeted to accomplish 
a specified work package in a specified time. 

b. Budgeted Cost of Work Performed (BCWP) 

The BCWP is the cumulative dollar amount of the sum of the completed 
work and the open work packages. The BCWP is the baseline item in cost reports. It 
represents what portion of the work has been accomplished in dollars [Ref. 30], As used 
in this research, the BCWP is the cumulative dollar amount of the projected costs of the 
work scheduled at a specified completion point. BCWP information is drawn from the 
DAES database. 

c. Actual Cost of Work Performed (ACWP) 

ACWP is the actual costs incurred and recorded in accomplishing the work 
performed within a specified time period [Ref. 6], As used in this research ACWP is the 
cumulative dollar amount of the actual costs for the work accomplished at a percentage 
completion point of the program ACWP information is drawn from the DAES database. 

d. Budget at Completion (BAC) 

The BAC is the dollar amount of the contract in terms of the cumulative 
cost of the BCWS. It is the contractually specified dollar amount. 

e. Contract Budget Base (CBB) 

The CBB is the negotiated contract cost plus estimates of authorized but 
unpriced work. The CBB includes Management Reserve, which is money reserved for 
within-scope changes to the contract. 

f. Estimate at Completion (EAC) 

EAC is the contractor's estimate of the cumulative cost at completion of 
the sum of all work packages. EAC in its simplest form is the dollar amount of all costs 
incurred to date plus the dollar amount of all costs remaining to complete the contract. 

The EAC may be derived by: 

EAC = ACWP + (B AC - BCWP) (2.1) 
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g. Cost Variance (CV) 

The CV is the difference between the dollar amount that was planned for 
the work package and the actual cost for the accomplished work package. It is 
determined by: 

CV = BCWP - ACWP (2.2) 

h. Percent Cost Variance 

The percent cost variance is a convenient means of comparing the 
magnitudes of differences between budgeted costs and actual costs. This research uses it 
as a comparison between different programs of varying budgets and schedules. It is a 
useful analysis tool and is given by: 

% CV = CV/BCWP (2.3) 

i. Cost Performance Index, Efficiency (CPI (E)) 

The CPI (E) measures the efficiency with which work has been 
accomplished by comparing the ratio of the budgeted cost of work performed to the actual 
cost. It is given by: 

CPI (E) = BCWP/ACWP (2.4) 

j. Cost Performance Index, Performance (CPI (P)) 

The CPI (P) is simply the inverse of the CPI (E). This measures the actual 
cost of each planned dollar of work accomplished. It is given by . 

CPI (P) = ACWP/BCWP (2.5) 

5. Estimates at Completion 

The estimate at completion (EAC) can be derived a number of ways. Some of the 
more common methods involve using an efficiency index of past work performed and 
extrapolating this into the future to determine the final cost of the project. The Cost 
Performance Index (CPI) is a commonly used tool to estimate the final cost. Three 
methods for obtaining CPI estimates for completion costs are: 

a. Cumulative CPI (CPIcum) 

The EAC (CPIcum) is given by: 

EAC (CPIcum) = BAC/CPI(E) 


( 2 . 6 ) 



This EAC uses the cumulative budgeted costs to date and the cumulative actual 
costs to date to derive an efficiency of work index. The assumption is the efficiency index 
derived will remain stable into the future. Gary Christie and Wayne Abba examined over 
500 contracts from the DAES database and found that the Cumulative CPI will remain 
stable between 15 and 85 percent complete in a program. Christie and Abba's 
observations were verified empirically by Heise [Ref. 17]. Heise examined a sample of 
155 contracts from the DAES database and confirmed that the Cumulative CPI remains 
stable between 20 percent complete and 85 percent complete. 

b. Weighted CPI 

This method of estimating the final cost of a contract is essentially the same 
formula as (2.6) above. The primary difference is the use of more recent data in the CPI 
(E) divisor. Commonly used weightings include 3 and 6 month CPI measurements. It has 
been found that the Weighted CPI will tend to give a higher estimate at completion than 
the Cumulative CPI. Normally, a range of estimates will be given to the program manager 
by the cost analyst. 

c. CPI/SPI 

The CPI/SPI estimate at completion is given by: 

EAC (CPI/SPI) = BAC x [(CPI(P)/SPI(E)] (2.7) 

The CPI/SPI estimate is based on a ratio of the cost of the work performed and the 
scheduled time of the work performed. This approach balances the cost and schedule to 
determine the ending cost. 

6. Cost Overruns 

Cost overruns are common in weapon system development programs. A Rand 
Report [Ref. 10] found the average cost growth in the development phase was 25 percent, 
and the average cost growth in the production phase was 18 percent. The report also 
found that overruns varied by Service. No single cause for the cost growth could be 
determined. Two variables that showed a strong correlation to cost growth were program 
size and program maturity. The smaller the program the greater the cost variance from 
the baseline, and the longer the program, the greater the cost variance. Table 1 provides a 
summary of the Rand reports findings. 
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Program Type 

Cost 

Growth 

Summary 

Cost 

Growth 

Vehicles 

1.77 

Total RDT&E 

1.25 

Aircraft 

1.28 

Procurement 

1.18 

Electronics 

1.24 



Munitions 

1.22 

Army Programs 

1.35 

Missiles 

1.17 

USAF Programs 

1.20 

Space Systems 

1.16 

Navy Programs 

1.16 

Helicopters 

1.13 




Table 1. Cost Growth by Type of Program. [Ref. 10] 

Surprisingly, the Rand report found that vehicles experience the greatest cost 
growth. The Rand study adjusted the budgets for inflation. The report did not test for 
statistical difference between either the types of programs or Service. Measuring cost 
variances is a relatively straightforward procedure. Estimating costs at completion is more 
difficult. The next section introduces a method that can provide estimates of the 
completion costs of a EMD program. 

D. THE RAYLEIGH DISTRIBUTION METHOD 
1. Overview 

The accuracy of a cost estimate is critical to the use of any method as a tool. 
Expenditures in developmental programs tend to follow a distinctive curve, often referred 
to as the Rayleigh Curve. This curve can be used to model natural events such as the 
growth of biological systems. Developmental projects tend to pattern this same type of 
growth during the lifecycle of the project. There are natural build-ups and build-downs in 
the level of resources consumed. 

Norden used this curve in Useful Tools for Project Management , to model the 
costs and resource use of software development. Abemethy found the curve could 
accurately model the cost expenditure pattern of weapon system programs. Elrod 
examined the application of the curve in EMD programs using two different models and 
found that it fit. Lee, Hogue and Gallagher developed a technique using the Rayleigh 
Function to estimate Research and Development budget profiles. Gallagher and Lee 
developed a technique based on the Rayleigh model to provide probabilities for the 
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estimate at completion. Substantial prior work has been accomplished that indicates the 
Rayleigh Function models actual expenditures well. 

Here the Rayleigh model is used to estimate the ending costs of the A-12 program, 
had it continued to completion in EMD. Figure 2 provides a graphical representation of 
the cumulative expenditure curve that models developmental program expenditures. 



Figure 2. Typical "S" Shaped Rayleigh Curve. [Ref. 18] 

2. Formulae 

The general form of the Rayleigh cumulative density function is given by: 

F(t) = l-exp(-at 2 ) (2.8) 

Where a is a time-scale parameter and t is the time period. [Ref. 18] Equation 
(2.8) is adapted for use in developmental programs in the following function: [Ref. 18] 

V(t) = d[l-(exp(-at 2 )] (2.9) 

Where V(t) is the earned value at time t. d is a positive constant that is equal to 
the total budget of the project (D) divided by .97. [Ref. 18] 

D = V(t f ) = ,97d (2.10) 

a is a constant that for EMD projects has been estimated to equal: 

a = 3.5/t 2 f (2.11) 

With t f equal to the final time of the project. [Ref. 18] 

Often the final time of the project is not known with certainty. If the time of peak 
rate of expenditures is known or estimated, then a can be estimated by: 

a = l/(2t 2 p ) (2.12) 
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Where t p is equal to the time of peak rate of expenditures. The time of peak rate 
of expenditures assumes added importance for programs such as aircraft or missiles which 
have scheduled first flight dates. It has been found that the peak rate of expenditure often 
occurs at the time of first flight. By taking the first derivative of equation (2.9) and setting 
it equal to zero, the time of peak expenditure rate, or first flight can be found. [Ref. 18] 

V'(t) = 2at[exp(-at 2 )] (2.13) 

Algebraic manipulation of equation (2.9) yields a formula for estimating the costs 
at completion, given a current ACWP at a specified time: [Ref. 18] 

d =[ V(t)/(l-exp(-at 2 ))] (2.14) 

E. APPLICATION OF THE RAYLEIGH FUNCTION TO THE A-12 PROGRAM 

1. Overview 

Early in this research the author realized that an appropriate tool for estimating 
costs at completion was needed. While common methods for estimating costs at 
completion involve the use of efficiency indices, this method was found to be 
inappropriate for estimations of completion costs when the CBB is subject to future 
changes. The CPI technique is based on the current budget and its estimates will be biased 
in this regard. The Rayleigh technique provides an estimate for what the future 
completion cost will be independent of current budgets. 

2. Justification 

Abemethy's previous work demonstrated a high correlation in the model's ability 
to provide ACWP estimates, given the final cost. Abemethy, concluded that while the 
Rayleigh Function could effectively model ACWP, given final actual costs, it was not a 
good predictor of final costs [Ref. 2], The question then becomes is the Rayleigh method 
a better predictor of final cost than the CPI methods? 

The Rayleigh Function was applied to a sample of 10 contracts using equation 
(2.14) to determine if the estimates at completion were closer than estimates using 
efficiency indices. An estimate at completion for each percentage completion point was 
computed using the Rayleigh Function, the Cumulative CPI, and a Weighted CPI. The 
total variance from the actual estimate at completion and the Rayleigh estimate showed an 
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accurate 


absolute difference much less than the two other methods, indicating a more 
model. 

Further refining the model by iteratively adjusting the shape parameters and 
establishing time bracket parameters resulted in even less variance in the Rayleigh 
estimates. The Rayleigh technique was able to achieve results that were 300 percent 
better than the Weighted CPI method and 200 percent better than the Cumulative CPI 
method. Elrod varied the alpha value of the model to determine a best fit to the data. In 
this research the alpha value is fixed at .00035 and the power of t from equation (2.14) is 
allowed to vary. The alpha value is able to be fixed at .00035 by standardizing all 
contracts to the same time period. This was accomplished by using percent completion 
points instead of months. Therefore every contract's time period consists of 100 and a 
remains constant at .00035. 

To optimize the model to the data the alpha value can not be a constant. This was 
adjusted for by allowing the constant power of 2 for the power of (-at) to vary. In effect 
equation (2.14) becomes: 

d = [V(t)/(l-exp(-at’ 1 ))] (2.15) 

Equation (2.15) was used to estimate the final cost of the A-12 Program. This 
equation varies from the original Rayleigh function by the addition of the variable x. The 
basis of the estimating technique remains equivelent to the Rayleigh methodology. 

The constant power 2 was replaced by the variable x to allow the model to adjust 
based on the percentage completion the program had achieved. During the research 
hundreds of tests were conducted using the Rayleigh model and actual contract data. It 
was found that the Rayleigh model consistently overestimated the completion costs early 
in the program and late in the program. By fixing a and allowing the power of t to vary 
according to the percentage complete a significantly more accurate model was obtained. 

In the research the variable x was allowed to assume one of ten values corresponding to 
percentage complete brackets. The addition of this third variable to the function changes 
the essential characteristics of what is considered the Rayleigh model. A full examination 
of the effects of this addition are outside the scope of this research. 

Gallagher refined the Rayleigh model using Multiple Model Adaptive Estimation 
(MMAE) [Ref. 14] to provide Bayesian statistical probabilities of outcomes for costs at 
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completion. By refining the model in this method it provided cumulative probabilities of 
the Rayleigh produced completion costs. Gallagher's MMAE model is used in this 
research to provide cumulative probabilities of occurrence of the EAC for the A-12 
program. 

F. THE BETA DISTRIBUTION 

Information on cost variances was collected from 58 developmental contracts. 
Based on the sample and the hypothesis to be tested, data from all 58 contracts were not 
used. The requirements that the data had to meet for each research question are contained 
in Chapter IV, Data. 

The sample data did not approximate a normal distribution. In this research a cost 
overrun is considered a negative cost variance while a cost underrun is considered a 
positive cost variance. The distribution of these data was skewed to the left, with the 
mean close to zero and data points extending far into the left tail. 

Using Best Fit [Ref. 20], the distribution that was most representative of cost 
overruns was the Beta distribution. The Beta distribution can assume many shapes based 
on the two parameters (v, w) that determine its shape. Figure 3 is provided to give the 
reader some insight into the degree to which the Beta distribution modeled the data. More 
information on the distribution of the data is contained in Appendix B. 
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Figure 3. The Beta Distribution. 
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G. SUMMARY 

This chapter provided an overview of the Acquisition Process and the C/S CSC 
terminology that will be used throughout this paper. 

The technique used for obtaining EACs for the A-12 program was presented 
While there are endless techniques for estimating completion costs, the Rayleigh model is 
used based on prior research and its ability to estimate costs independent of current 
budgets. 

Cost variances are usually negative. The distribution of cost variances for the 
sample was skewed to the left. The Beta distribution tended to model cost overruns better 
than the Normal distribution. A graphical presentation of the Beta distribution is helpful in 
picturing the distribution of cost overruns 

Chapter III completes the background portion of this paper. Chapter III examines 
the A-12 Program and the cost variances that caused the termination of the Program. 
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III. THE A-12 PROGRAM 


A. OVERVIEW 

This chapter is intended to give the reader background information on the Navy's 
A-12 development program. As shown in Chapter II, cost overruns in defense 
acquisitions are not unusual. This chapter supports the primary research question by 
examining the A-12 Program and the cost overruns that occurred. For a more in-depth 
view of the A-12 Program the reader is referred to The Beach Report [Ref. 2], the Navy's 
internal review of the Program that was conducted in 1990. 

B. MISSION NEED 

The A-12 Avenger was designed to replace the A-6 Intruder in the Navy's carrier 
based fleet. Its projected survivability and adaptability would allow the Navy to also 
replace the EA-6B, Advanced Tactical System aircraft. The Navy also considered the air- 
to-air capability of the A-l 2 in possibly replacing the F-14s in the fleet defense role. The 
various roles that the A-12 could fill would provide a more streamlined logistical system 
by reducing the various spare parts that would be carried in inventory [Ref. 21], The 
aircraft was to be a carrier-based medium range attack aircraft with low observable 
(stealth) characteristics. The United States Air Force (USAF) was interested in procuring 
up to 400 A-12 variants that would not require the structural hardening to land on aircraft 
carriers. The Air Force's version was planned to be the replacement for the F111. The Air 
Force planned on taking initial delivery in 1995. [Ref. 9] 

The first version of the A-6 entered the fleet in 1963 as an all weather, day/night 
medium attack aircraft. The latest version of the A-6, the A-6E, was introduced in 1972 
and procurement continued through 1987. In the 1980's wing cracks began appearing in 
the inventory of the A-6s. The stress of landing on carriers required that when the wings 
reached 67 percent of their life, the plane was limited to restricted duty until the wings 
were replaced. By 1988, most of the inventory of A-6s was rapidly degrading In 1988, 
the year the contract for the Full-Scale Development of the A-12 was awarded, the Navy 
placed its last production order for the A-6 The A-12, with an Initial Operational 
Capability (IOC) of 1994, was to replace the A-6s. The first squadron was projected to be 
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combat ready by late 1996 [Ref. 21]. Figure 4 shows the projected A-12 aircraft 
superimposed over the F-14 and the A-6. 



Figure 4. Aircraft Comparison. [Ref. 21] 


C. EARLY DEVELOPMENTAL WORK 

In 1984 Concept Exploration work began on the A-12 Program. The F-l 17 
Stealth Fighter proved the validity of stealth technology. The F-l 17 was operational for 
one year with the USAF in the Nevada Desert, however, it was still tightly classified. The 
early development work on the A-12 was also classified as a special access program. 

In 1984, two separate contractor teams began concept exploration for the new 
aircraft. This Program was initiated in 1984 with the start of Concept Exploration. In 
June 1986 the same contractor teams were awarded contracts to continue work into 
Demonstration and Validation. In June 1986, Captain Lawrence Elberfeld was appointed 
as the Program Manager of the Program which was now in Phase I of the Acquisition 
Process. Captain Elberfeld participated in the source selection for the A-12. Phases 0 and 
I of the acquisition cost $2.8 billion. [Ref. 25] 

D. FULL-SCALE DEVELOPMENT 

1. The Contract 

The FSD contract was awarded to McDonnell Douglas/General Dynamics in 
January 1988. The contract was a Fixed-Price-Incentive-Firm type contract with a share 
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ratio of 60/40 above contract target cost. The target price was $4,379 billion, the target 
cost was $3,981 billion and the ceiling price was $4.77 billion [Ref. 9], On any overruns 
above target cost, from $4.4 billion to $4.8 billion, the Government would pay 60 cents of 
every dollar and the contractor would pay 40 cents. The Point of Total Assumption was 
$4.65 billion. Progress payments to the contractor were approved at the 80 percent rate. 

The contract required the full-scale development and prototype production of the 
aircraft. The original first flight was scheduled for June 1990. This date was adjusted 
several times throughout contract performance, until just prior to termination the 
projected first flight date was March 1991. The time of first flight in an aircraft 
development contract is of crucial importance in estimating cost expenditures. As was 
explained in Chapter D in more detail, the point of first flight is usually the point of peak 
expenditures of the program. Given this point the total cost of the contract at completion 
may be estimated. Historical data indicate that 40 percent of the development costs are 
expended at the point of first flight [Ref 14], The impact of delaying the first flight results 
in increased cost and extended development schedule [Ref. 18], Table 2 provides the 
adjustments made to the scheduled first flight of the aircraft over the performance of the 
EMD contract. 


March 1991 
December 1991 
June 1992. 


November 1989 
March 1990 
May 1990 
November 1990 
December 1990 


Contractually Specified 
Nov 89 DAES report 

Contractor Brief to MAR 
Steering Committee 
Contractor Brief to PM 

Contractor request to 


Table 2. A-12 First Flight Adjustments. 

The contract also required delivery of "eight flight test aircraft and five full-scale 
ground test articles..." Additional provisions of the contract provided for three production 
option lots of 4, 6, and 16 aircraft, and an option for a fourth production lot at the 


29 






completion of the Critical Design Review. The first two production lots were for pilot 
production and the third and fourth lots were low rate initial production. [Ref. 9] 

2. Costs 

Initial cost estimates were based on the purchase of 858 total aircraft at an annual 
procurement rate of 48 aircraft. With the reduction of the fleet to 14 aircraft carriers the 
requirement for A-12s dropped to 620 aircraft purchased at an annual rate of 36. 

The cost of the A-12 was initially projected to be $86.6 million per aircraft, given a 
production rate of 48 per year. When the requirement dropped to 36 aircraft per year, the 
cost per aircraft increased to over $100 million per aircraft. The Lot I pilot production 
was priced at $1.2 billion for 6 aircraft [Ref. 24], As production continued and the 
quantity produced increased with the anticipated USAF demand, prices were expected to 
fall, according to Learning Curve theory. 

A graphical cost comparison of the A-12 to other combat aircraft is shown in 
Figure 5. Costs shown are adjusted to 1995 dollars using Department of the Navy 
inflators. All data are drawn from U.S. Weapon Systems Costs [Ref. 5], 
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Figure 5. A-12 Cost Comparison. [Ref. 5] 

The cost data presented are from 1975 to 1995, and adjusted to 1995 constant 
dollars using Department of the Navy inflator values. As shown, the cost of the A-12 
would have been significantly higher than other aircraft in the inventory. The cost data for 
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the F-l 17 are still restricted, so no comparison can be made to other stealth fighter 
aircraft. 

3. Costs at Completion 

The decision to terminate the A-12 Program was based on the cost overruns that 
the Program was experiencing and the uncertain estimate of costs at completion. The 
Secretary of Defense stated that the inability of being able to determine the final cost of 
the Program was a primary reason for the termination. 

The cost at completion is driven by the first flight date. As the programmed first 
flight of the aircraft is extended, total program time is extended and the costs increase. 
The difficulty of determining a final cost is highlighted in Table 3. Table 3 provides final 
cost estimates of the Program determined by the Department of Defense Inspector 
General (DoD IG). 



The DoD IG estimates were prepared by the Cost Analysis Improvement Group 
(CAIG). The CAIG estimates provide EACs based on three possible first flight dates and 
three possible percentage completion points. The June 1991 first flight date is the most 
optimistic date. The range of values is based on assumptions of the percentage of the 
programs that would be complete at first flight, with 60 percent being the most optimistic. 

4. The Contractors 

The solicitation for the A-12 required a teaming arrangement. A teaming 
arrangement is intended to provide two qualified contractors for production to reduce unit 
costs. When the FSD phase of the A-12 contract began 10 U.S.C. 2438 required the 
Department of Defense to establish at least two competitive sources for production. This 
law was later amended in November of 1990. [Ref. 23] 
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The purpose of a teaming arrangement is to avoid reliance on a sole source of 
production for the system. Each contractor is to provide his particular area of expertise 
for the program and have the capability to independently produce the system after 
development. In the A-12 Program, the contractors established a written agreement that 
provided each team member would try to convince the Navy to guarantee a minimum of 
40 percent of the production to each team member. When the requirement for the A-12s 
was reduced to 620 aircraft a competitive production contract was not feasible. 

According to a General Accounting Office (GAO) report issued in 1992, ". ..an 
award split other than 50-50 or 60-40 would have put the losing contractor out of the 
A-12 business." [Ref. 23: p. 7]. 

The larger member of the A-12 team, McDonnell Douglas, was created in 1967 
with the acquisition of the financially troubled Douglas Aircraft Company by McDonnell 
Company. McDonnell Douglas has been the Nation's top prime contractor in all but one 
of the years from 1984 to 1994 [Ref. 8], Its average share of the Department of Defense's 
prime contracts has averaged around 6 percent of the value of total contracts awarded. 
According to Financial World magazine McDonnell Douglas has a market value of $5.6 
billion, 154th in value of all U.S. companies. In 1994 McDonnell Douglas had $13.3 
billion in sales and $562 million in profits [Ref. 13], 

McDonnell Douglas produces the F-15 Eagle, F-18 Hornet, C-17 Globemaster III, 
AV-8 Harrier, T-45 Goshawk, AH-64 Apache, and the Tomahawk and Harpoon missiles. 

It also has RDT&E contracts for electronics and communications equipment, along with 
missile and space systems. [Ref. 5] 

The other member of the A-12 team, General Dynamics, has consistently been in 
the top ten Defense contractors for the past ten years. Its dollar value of prime contracts 
has ranged from $2 billion to $8 billion. Its market value is $2.75 billion, placing it at 
number 329 of the nation's top companies. In 1994 it had $3 billion in sales and $214 
million in profits. [Ref. 13] 

General Dynamics produces the F-16 Falcon, the Seawolf nuclear submarine, M-l 
tank. Tomahawk missile, Stinger missile, RIM-66 missile and the MK-15 Close in Weapon 
System. [Ref. 5] 
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Both companies have a history of litigation actions with the Government. They 
currently are contesting the $1.35 billion in unliquidated progress payments demanded by 
the Government and they have a certified claim for equitable adjustment for $1.4 billion 
resulting from the termination of the A-12 Program [Ref. 28, July 27 1994], A ruling in 
December 1994 that vacated the Department of Defense's Termination for Default was 
considered a victory by the contractors in their efforts to clear their financial debt to the 
Government on the A-12 Program. 

E. TERMINATION 

1. Overview 

The A-12 Program was terminated for default on 7 January 1991 The Secretary 
of Defense directed the Navy to terminate the troubled Program. Although many reasons 
contributed to the termination of the Program, this research is primarily concerned with 
the stated reason for termination: cost overruns. The estimate at completion for the A-12 
varied significantly. The Secretary of Defense stated uncertainty in estimating the cost of 
the contract was the primary reason for terminating the Program. 

This program cannot be sustained unless I ask Congress for more 
money and bail the contractors out, but I have made the decision that I will 
not do that. No one can tell me exactly how much more money it will cost 
to keep this program going. I do not believe a bailout is in the national 
interest. If we cannot spend the taxpayers' money wisely, we will not 
spend it. [Ref. 22: p. 3] 

2. Costs at Termination 

The cost of the work accomplished in the A-12 Program is difficult to estimate 
accurately. Under a fixed-price type contract the contractor is required to provide a 
deliverable to the Government. The Government will pay for the products or services it 
has accepted. 

In the A-12 Program, the value of the work accomplished was priced at $1.34 
billion. The contractors had incurred much greater costs in the development of the 
aircraft. Table 4 provides a detailed accounting of work accepted by the Government at 
termination. 
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ITEM 

Price 

(In Millions) 

Initial Design Review 

269.8 

Preliminary Design Review 

261.9 

Critical Design Review (Engine) 

139.0 

Critical Design Review (Phase IA) 

150.6 

Phase 1A Test Review 

197.0 

Program Management Review 

3163 

Total 

$1,334.7 


Table 4. Value of Work Accepted. [Ref. 26] 

While the value of work accepted was priced at $1.33 billion, the actual price paid 
for the work was $2.69 billion. Of this $2.69 billion the Government has demanded 
$1.352 billion in unliquidated progress payments be returned. The contractors are 
currently disputing the decision of the Government of the value of work accepted. [Ref. 
25]. 

Total expenditures on the A-12 Program exceeded the value of the work accepted 
at termination. Table 5 provides a listing of expenditures on the A-12 Program at the 
point of termination. 


CATEGORY 

$ 

(Billions) 

Total Appropriations 

6 7 

Total RDT&E Expenditures 

2.85 

FSD Contract Expenditures 

2.58 

Long Lead Procurement 

0.104 

Value of Work 

1 34 

Unliquidated Progress Payments 

1.35 


Table 5. A-12 Program Expenditures. [Ref. 26] 


3. Contractor Claims 

The contractors have an immense financial interest in the favorable settlement of 
the litigation. The conversion of the termination, from default to convenience, would 
completely restructure the settlement amounts. 

The contractor team has filed a claim for equitable adjustment of $1.4 billion. The 
contractors claimed that the contract could not be completed according to the 
contractually specified terms because of: 
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1. The Navy's superior knowledge of facts vital to performance. 

2. Delays and disruptions caused by the Navy's conduct. 

3. The Navy's flawed acquisition strategy. 

4. Commercial impossibility of performance. 

The $1.35 billion demanded by the Government has been deferred pending the 
outcome of the litigation. [Ref. 25] 

F. THE JOINT ADVANCED STRIKE TECHNOLOGY PROGRAM 

1. Overview 

The Navy committed seven years and $2,985 billion to the A-12 Program. If the 
contractors are successful in their litigation the cost of the A-12 Program to the 
Government could go as high as $6 billion. At the termination of the Program the Navy 
had very little that it could use in a future development program. During hearings by the 
Committee on Armed Services of the House of Representatives on the A-12 termination. 
Congressman Sisisky called the termination of the A-12 Program " . .a national disaster 
with naval aviation." 

2. The Next Fighter 

The cost of developing a fighter aircraft to fulfill just the Navy's needs is 
prohibitively expensive. A joint program with the Air Force is cunently ongoing called 
the Joint Advanced Strike Technology Program (JAST). 

The JAST aircraft is being designed as a low observable technology aircraft that is 
projected to replace the Navy's A-6, the USAF's F-16 and the Marines AV-8. While the 
JAST aircraft will not have the ordinance carrying capacity that was provided by the A-6, 
this shortcoming is expected to be made up for in its use of smart weapons and increased 
survivability. 

The JAST Program started in 1994 and is currently in Phase 0 of the acquisition 
process. It is expected to start EMD in the year 2000, Low Rate Initial Production 
(LRIP) in 2005, and Initial Operational Capability by the year 2007. [Ref. 16] 

The total cost (life cycle cost) of the Program is expected to range from $160 to 
$200 billion. Unit procurement costs are expected to be from $30 to $40 million. This 
figure is relatively inexpensive when compared to the cost of other currently operational 
combat aircraft (See Figure 5). 




The JAST aircraft is being conceived of as a modular aircraft that will utilize a 
common fuselage and Service unique components would then be added on. According to 
the Program Office, 85 percent commonality of parts is the goal. [Ref. 16] 

The Mission Need Statement and the Operational Requirements Document from 
the A-12 Program were used as the base for developing the MNS and ORD for the JAST 
Program. The Program Manager is an 0-7. The duties of Program Manager will rotate 
between the USAF and the Navy. [Ref. 16] 

G. SUMMARY 

Chapter III provided a short summary of the A-12 Program. The Program 
generally followed the basic Acquisition Process outlined in Chapter II. Although the 
A-12 was a restricted access program it was still subject to oversight. Shortly after the 
start of the FSD contract the Program started incurring cost overruns. As the Program 
continued the overruns became worse. The uncertainty in the estimates of final 
completion costs resulted in the termination of the Program. 

Chapter IV presents comparison data of other EMD programs and cost variances. 
These data are used to contrast the A-12 cost variances and determine if the A-12's 
overruns were different than other major acquisitions. 
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IV. DATA 


A. OVERVIEW 

This chapter provides the data that are used to answer the primary research 
question: Was the A-12 Program over cost when compared to other similar acquisitions? 
Chapter IV begins by providing the actual reported cost data for the A-12 Program up to 
the point of contract termination. Data from 58 comparison contracts are then provided 
as the sample to be tested against. 

There are hundreds of contracts in the DAES database. The selection of the 
contracts used in this research conformed to the requirements set out in Chapter I, with a 
few exceptions. Sixty-five EMD contracts were initially drawn from the database. Seven 
contracts were eliminated from the sample due to excessive rebaselining or insufficient 
data points. Three contracts were retained that did not meet the ten data point 
requirement. These contracts were for aircraft development and met the other specified 
requirements for inclusion. 

It became apparent, as this research was conducted, that all programs experience 
adjustments to the Contract Budget Base (CBB). In the 58 contracts examined for 
comparison with the A-12 Program only two contracts showed a downward adjustment in 
the CBB. In one case the program's funding was reduced by nearly half, the other 
contract showed a cost at completion that was just slightly under the original estimate. 

The effects of inflation do not account for the upward revisions. Contractor 
expenditure budgets are based on future predictions of price escalation. This was 
confirmed by a former program manager of Lockheed Corporation and a Cost Analyst 
with the CAIG. The explanations for the adjustments to the CBB can include poor 
original understanding of the scope of work, or adjustments to the contractual 
requirements. Regardless of the cause, only five of the 58 contracts examined finished the 
EMD phase within five percent of the original CBB. 
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B. A-12 PROGRAM COST DATA 


1. Program Information 

The A-12 Program began the EMD phase in January 1988. Significant effort on 
the Program was expected to be complete by April 1996, with first flight of a prototype 
aircraft in June 1990. During the life of the A-12 Program, 27 Cost Performance Reports 
and nine Selected Acquisition Reports (SAR) were submitted. The data from the A-12 
Program will be compared to the sample 58 contracts to determine if the A-12 cost 
variances were within the population of cost variances. 


The comparison between the A-12 cost variances and the sample is based on cost 
variance at several percentage completion points. Table 6 provides the reported A-12 cost 
information. 


report % % 

DATE _ BCWP _ ACWP _CV_ COMPLETE _ CBB 

Sept 1988 303.2 325.6 -7.39 7.6 3981.1 


Dec 1988 441.7 494.5 

March 1989 594.6 689.4 

Sept 1989 1009.6 1220.5 

Dec 1989 1279.2 1597.1 

Apr 1990 1491 1950 

June 1990 1704.4 2285.5 

Sept 1990 1881.5 2611 

Nov 1990 1990.6 2789 


Note: Figures are reported in millions of constant dollars. 


-20.89 25.4 3982.2 

-24.85 31.6 4042.3 

-30.78 36.8 4046.4 

-3499 42;1 4047.1 

-38.77 46.5 4044.7 

•40.11 49.2 4045 


Table 6. A-12 Reported Cost Information. 

The A-12 cost data are drawn from the nine submitted Selected Acquisition 
Reports. The percent complete and the percent cost variance are as reported. The 
Program was terminated in January 1991. The accuracy of the reported cost data is of 
great importance in determining whether the A-12 Program differed significantly from 
other similar programs. 
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2. Accuracy of Reported Cost Data 

The accuracy of the A-12 ACWP data can be affected by numerous factors. There 
is delay from the time work has occurred to when it is reported. There can also be 
material understatements of the actual cost of the work performed. 

During the EMD phase of the A-12 Program, McDonnell Douglas was 
experiencing cash flow problems [Ref. 15], Significant cost variances reduce the amount 
of the progress payments based on estimates at completion. A reduction in the amount of 
the progress payments made to McDonnell Douglas would have had a negative financial 
impact in what was already a poor financial climate for the contractor. Incentives exist to 
under report cost information on an over-budget program, or to shift costs from an 
over-budget program to a within-budget program. 

Evidence of McDonnell Douglas shifting costs from the A-12 Program to other 
Government contracts was reported in the September 11, 1995 edition of the Wall Street 
Journal. According to the article, the U.S. Justice Department has decided to litigate a 
whistle blower case. It is alleged, "...company supervisors systematically ordered 
assembly-line employees to improperly shift untold millions of dollars of labor costs 
between various weapons programs, including the C-17 cargo plane, F-18 fighters and the 
Navy's now-cancelled A-12 carrier based stealthy attack jet." The article further states 
that the C-17 and the A-12 were "the biggest beneficiaries of the alleged cost shifting." 
The amount of money involved is in the hundreds of millions of dollars. [Ref. 28, 11 Sept. 
1995] 

No correction can empirically be made to the actual reported costs for the 
Program. If the ACWP figures have been understated it can not be shown within the 
bounds of this research, and the impact it may have had on the Program is also out of the 
scope. An assumption is made that cost shifting may occur in any given program and no 
adjustment is made to the data. 

3. The Problem of Rebaselining 

The percentages complete of the A-12 Program as shown in Table 6 is based on a 
Contract Budget Base of $3.98 to $4.05 billion. Since the contract would have exceeded 
the ceiling price of $4.7 billion the Program would have been rebaselined if it had 


39 




continued to completion. No significant adjustments were made to the CBB in the A-12 
Program. In all the comparison contracts some degree of rebaselining occurred, in many 
cases the amount of the rebaseline was quite significant. Table 7 shows the average 
percentage adjustments in the sample program's CBBs. For contract information, the 
reader is referred to Appendix A. To determine the CBB adjustments a relatively large 
percentage of the program had to be complete. The percentage was determined to be data 
points covering at least 50 percent of the total program, and a program completion point 
achieved of at least 70 percent. The adjustment is determined by measuring the difference 
between the initial CBB and the ending CBB. 



Adjustments to the CBB have two major affects on the reported cost information. 
First, the percentage completion is reduced based on the new budget. Second, the cost 
variance percentage changes, based on the new BCWP figures. 


Both affects impact the ability to estimate final completion costs accurately. 
Adjustments to the contract's budget can be made for a variety of reasons. The more 
stable the baseline, the more accurate the estimate at completion will be. 


Table 8 provides adjusted percentage completion points of the A-12 Program 
based on three time estimates to complete. 



Table 8. Rayleigh Model Estimates of A-12 Program Expenditures. 
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The first time estimate is the contractually specified first flight and contract 
length. The second and third estimates are based on estimates of the first flight date, at 
contract termination. The first flight dates were obtained from Reference 9. The first 
flight date, in months from program start, is multiplied by 2.65 [Ref. 18] to estimate the 
total program length. The estimates at completion were derived using the Rayleigh model 
methodology described in Chapter II. The percentage complete at termination is 
determined by taking the termination cost of the A-12 and dividing it by the EAC. The 
EAC is based on an assumption of 40 percent of costs incurred at first flight. 

4. The Estimate at Completion 

The accuracy of using the Rayleigh model to estimate cost at completion was 
tested by Elrod [Ref. 11], Elrod found a generally good fit between the Rayleigh model 
and actual costs at completion. Gallagher, a cost analyst in the Cost Analysis 
Improvement Group (CAIG), further developed this technique [Ref. 14]. 

The Rayleigh model's predictive capability was tested against ten aircraft program 
contracts. A comparison was then made using the CPIcum technique to determine the 
more accurate predictor. The results of this test are provided in a graph in Figure 6. 
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As shown in Figure 6, the Rayleigh estimates provide significantly less variance 
from actual completion costs over the life of the program. To derive the curves in the 
graph, data points from all ten programs were plotted against time and an exponential 
curve was fit to the data. Generally, past the 20 percent completion point, the Rayleigh 
model provides reasonable estimates of actual program completion costs. The percentage 
cost variances are within 10 percent of actual ending costs at the 40 percent completion 
point. Using the CPIcum technique, the estimates are not within 10 percent of actual 
ending costs until the 80 percent completion point. On the basis of Elrod and Gallagher's 
work, and the test against ten actual contracts, the Rayleigh model is accepted as 
providing a realistic estimate of A-12 Program costs. 

5. Likelihood of the Estimates 

An estimate of a future occurrence is by definition uncertain. It is of interest to be 
able to say with what probability the outcome will occur. The estimates of the final cost 
of the A-12 Program in Table 8 were tested using a software program developed and 
provided by Gallagher [Ref. 14] to determine the likelihood that the ending cost estimate 
would occur. Gallagher's work uses past cost figures to develop a Bayesian probability of 
future occurrence of the ending cost. He developed this technique while working as a cost 
estimator with the CAIG. 

Three estimates at completion were tested using Gallagher's model. The first 
estimate was the Program Manager's EAC at contract termination. The second and third 
estimates were the EACs produced by the Rayleigh model for first flight dates of June 
1992 and December 1992. Table 9 gives the probability that final ending costs would 
exceed the estimate. For a detailed explanation of the development of the probabilities 
and the Rayleigh estimates, the reader is referred to Final-Cost Estimates for Research & 
Development Programs Conditioned on Realized Costs by Mark A. Gallagher and David 
A. Lee [Ref. 14]. The article is available through the Defense Technical Information 
Center database. 
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1ST FLIGHT_PROBABILITIES OF EACs_ 

P(X>=4.7B) P(X>=7.5B) P(X>9.07B) P(X>=11B 

June 1992 0.99 0 84 0.63 0.37 

Dec 1992 0.99 0.87 0.70 . 0.49 

Table 9. Probability of Final Cost Estimates. 

In Table 9, the cumulative probabilities of occurrence of the given cost estimate 
are provided. The first column, 1st Flight, presents two possible first flight dates. These 
dates are drawn from the CAIG estimates provided to the DoD IG. Columns two through 
four provide four possible EACs and the probabilities of the Program exceeding the given 
EAC. The first EAC is the original ceiling price of the contract. The probability that the 
Program would have exceeded this ceiling is almost certain. If first flight would have 
occurred in December 1992, the probability that the Program cost would have exceeded 
$11 billion is .49. 

To compute the probabilities shown in Table 9, two time assumptions are required. 
For a first flight date of June 1992, the minimum Program length was assumed to be five 
years from initiation with the maximum length of 13 years. If the first flight would have 
occurred in December of 1992, then the minimum length was assumed to be eight years 
with a maximum length of 15 years. 

C. COMPARISON CONTRACT COST DATA 

1. The Comparison Sample 

Over 100 contracts were examined from the DAES database for inclusion into the 
sample. Sixty-five contracts were initially found to meet the requirements as specified in 
Chapter I: at least ten data points existed, the program had achieved a significant level of 
completion, and the program was classified as an ACAT ID program. Of the 65 contracts 
selected for comparison, five were eliminated due to non-consistency in completion 
percentages and two contracts were eliminated due to insufficient data. 

2. Characteristics of the Sample 

Fifty-eight contracts constitute the sample. To be included in the DAES database 
the program must be an ACAT ID program. All the contracts examined were for major 
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programs. The completion points of the sample ranged from a low of 40 percent complete 
to a high of 100 percent complete. The average completion point achieved by the 58 
sample contracts was 83 percent complete. Table 10 shows the characteristics of the 
sample examined. 


FIXED COST 

_ PRICE _ REIMB _ TOTAL 

ARMY 1 18 19 

AF 13 7 20 

NAVY 2 _ " , 17 . 19 ..... 

TOTAL 16 32 58 


Table 10. Characteristics of the Sample. 

3. Sample Data 

The cost variances of the sample were determined using equation (2.3). All 
contracts that met the specified criteria in Chapter I were used. Not all the sample 
contracts had data points for each of the percentage completion points. For those 
contracts that had multiple data points within a percentage completion range, the cost 
variances were averaged to avoid weighting data from any one contract more than other 
contracts. 

The data for the percentage completion brackets are each modeled seperately. 

Only the 0 to 10 percentage completion bracket exhibits the Normal distribution that one 
would expect if cost variances were distributed about a mean cost variance of zero. 

Table 11 provides the cost variances observed in the sample contracts. In Table 
11, the completion point refers to the percentage complete the program has achieved 
measured by time expended divided by total time. Column 2 gives the arithmetic mean of 
the sample. The data are skewed to the left. The median is provided as a reference point. 
The sample of all 58 contracts is modeled by the Beta distribution. Column 4 indicates the 
distribution that modeled the data using a Chi-squared test. 
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Table 11. Cost Variances of Sample Contracts. 


4. Aircraft Program Data 

Seventeen contracts were included in the aircraft cost variance sample. Three 
contracts for aircraft developmental work were included in the sample that had only nine 
data points. To be included in the sample the program had to show relatively increasing 
degrees of completion as work progressed. The aircraft program contracts consisted of 
the contract for the program itself or for major aircraft type subassemblies. Contracts for 
subassemblies such as avionics, engines or armaments were not considered. Contracts for 
follow-on developmental work were also not included. Helicopters were considered as 


aircraft. 


Of the 19 contracts initially examined only one program was terminated. The 
remaining programs continued into production or are still in the EMD phase. The one 
program that was terminated was later restarted under a different designation. This 
contract was not considered for inclusion since many of the technical problems that caused 
the original termination would have been paid for and would not have shown up in the 
new ACWP figures. Table 12 provides the mean and median cost variances of EMD 
aircraft programs. 
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COMPLETION 

POINT 


MEAN 

CV 


MEDIAN 

CV 


0-10% 
11-20% 
21-30% 
31-40% 
41 50% 
51-60% 
61-70% 
71-80% 
81-90% 
>90% 
ALL 


-2 99% -4% 11 

-8.12% -5.18% 11 

-4 05% -2.51% 12 

-8 38% -5.34% 14 

-13.58% ; -8.17% 10 

-12.51% -8.99% 12 

-16.88% -815% 12 

-16.79% -11.33% 13 

-11 96% -6.09% 10 

-17.93% -9.01% 8 

-18.4% -6.13% 17 


Table 12. Aircraft Program Cost Variances. 

The mean cost variance for all aircraft programs examined was -18.4 percent. The 
median value was -6.13 percent. Seventeen programs were included in the sample. 

5. Fixed-Price Versus Cost-Type Contracts 

All contracts that had exceeded 50 percent complete were included in the sample. 
The mean cost variance for all programs examined was -16.5 percent. Table 13 provides 
comparison data for mean overruns of fixed-price type contracts versus cost-type 
contracts. 

TYPE MEAN MEDIAN 

CONTRACT _ CV% _ CV% _ N 

FP -21.02% -10.2% 15 

COST -14.76% -6.05% 39 

TOTAL -16.5% -6.78% 54 

Note: Total=Most recent cv for contracts > 50% complete. 

Table 13. Fixed-Price Versus Cost-Type Contract Cost Variances. 
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D. SUMMARY 

This chapter presented the data to be used in Chapter V to determine if the A-12 
Program was significantly different from other EMD programs. The difficulty of 
developing accurate estimates at completion is caused by several factors such as 
rebaselining, changes, or extensions of program length. The mean cost variance of EMD 
programs is -16.5 percent with a median value of -6.78 percent. While the median value 
for cost variances does not appear to be too significant, the true cost variances are masked 
by adjustments to the CBB. The mean adjustment to the CBB is 54 percent with a median 
value of 46 percent. 

Adjustments to the CBB and cost variances are evaluated in Chapter V. Chapter 
V tests the hypothesis that the A-12 Program cost variances were no greater than other 
EMD programs. The next chapter also examines secondary causes for the termination of 
the A-12 Program. 
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V. DATA ANALYSIS 


A. OVERVIEW 

Chapter IV presented the data from 58 separate EMD contracts. Chapter V 
establishes percentile values for the sample programs and compares the A-12 Program 
cost variances. Initially, the A-12 is compared to the entire sample for differences. The 
A-12 cost variances are then compared to other aircraft programs. 

B. A-12 COST VARIANCES 

1. Percentile Values 

To provide a valid comparison of the sample data and the A-12 cost variances, the 
cost variances of the sample are grouped by percentage completion points. Percentile 
values are determined for each sample point based on the type of distribution. The 
procedure to accomplish this was to use the statistical software program. Minitab [Ref. 
19], The Inverse CDF command returns the percentile value for the specified point. With 
the Beta distribution the critical value is a value from 0 to 1. This value is converted to 
the actual value using techniques described in Appendix B. Table 14 provides the 
percentile values for the designated percentage completion points. 
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The data from Table 14 are used to answer the primary research question and 
Hypothesis 1: Point values of the A-12 are no different from cost variances of the sample. 
The first column of the Table provides the percentage completion points of the sample 
contracts. Column 2 provides the distribution that best modeled the data, the resulting 
Chi-squared value is in Column 5. The parameters associated with the distribution are in 
Columns 3 and 4. For the Normal distribution, the parameters are given by the mean and 
standard deviation. The percentile values are the area of the distribution above the given 
value. 

2. A-12 Cost Variance Percentiles. 


After determining the critical values in Table 14, the next step was to evaluate 
where the A-12 cost variances were in relation to the distribution. Table 15 provides the 
comparison of A-12 cost variances versus the sample. 



Table 15. A-12 Cost Variance Percentile Values. 


The results from Table 15 agree with prior work by Heise [Ref. 18], Christensen 
[Ref. 3], and observations made by Abba and Christie. As the program proceeds the cost 
overrun does not improve. 
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In Table 15, Columns 1 and 2 are the percentage completion points and cost 
variance as reported. The distribution that modeled the data is in Column 3. The final 
Column provides the percentage of programs that the A-12 cost variance exceeded. 

The A-12 cost variances exceeded 75 percent of sample programs at the 7.6 
percent completion point. The cost variances worsened from that point on. Given Heise's 
earlier findings that the CPIcum will remain stable from the 20 percent completion point 
on, it is unlikely that the cost variances of the A-12 would have improved, if the program 
had continued. The relation of the A-12's cost variances to other programs did not 
improve. At termination the cost variance of the A-12 exceeded 97 percent of all other 
programs. As shown from the sample data, the A-12 cost overruns were exceptional. 

C. AIRCRAFT PROGRAMS 


1. Aircraft Program Percentile Values 

In Table 16, aircraft program cost variances are examined to determine if A-12 
cost variances differed from other aircraft programs. The same analysis techniques were 
used as with the comparison to the overall sample. 


21-30 

31-40 


17.62 

20.51 


-11,65 

-23.06 

-4254 


-13:42 

-27.29 


Table 16. Aircraft Program Percentile Values. 

The sample of aircraft programs includes 17 separate contracts. Cost variances 
are determined for each percentage completion bracket. The data are modeled using 
BestFit [Ref. 20], Appendix B contains complete information on rankings and selection of 
distribution types. Percentile values were determined using Minitab. 
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2. The A-12 Program Versus Sample Aircraft Programs 

The reported A-12 cost variances and percentage completion points were 
compared to other aircraft programs to determine differences. The results are provided in 
Table 17. 


AREA 

% % TYPE ABOVE 

COMPL CV DISTRO VALUE 


Beta 0.736 


7.6 -7.39 

11.1 -11.95 

14.9 -15.94 

31.6 -24.85 

36.8 -30.78 

42.1 -34.09 

46.5 -38.77 

49.2 -40.11 


Beta 0.7112 

Beta 0.7968 

Normal 0.9997 

Beta 0.9708 

Beta 0.9955 

Logistic 0i8921 

Logistic 0.9291 

Logistic: 0.9373 


Table 17. A-12 Versus Sample Aircraft Programs. 

The distribution that best modeled the data from the aircraft programs was used to 
compute the area under the cumulative density function (CDF) for the A-12 values. This 
was accomplished using Minitab and the CDF command. 

The area given in Column 4 is the percentage of programs that the A-12 cost 
variance exceeded. At 7.6 percent complete, the cost variance in the Program exceeded 

73.6 percent of other aircraft programs. At termination, the A-12 cost variance exceeded 

93.7 percent of other aircraft programs. The data indicate that the A-12 Program cost 
variances were exceptional when compared to other aircraft programs. 


D. FIXED-PRICE VERSUS COST-TYPE CONTRACTS 


1. A-12 Cost Variances Versus Contract Type 

This analysis is conducted to determine the degree to which the selection of a 
fixed-price contract may have effected the cost overruns in the A-12 program. The mean 
cost variance of a fixed-price contract is -21.02 percent. The median cost variance is 
-10.2 percent. Fifteen contracts were included in the sample. 
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The mean cost variance of cost-type contracts is -14.76 percent with a median 
value of -6.05 percent. Thirty-nine contracts were in the sample. Table 18 provides the 
results of a comparison between the A-12 cost variances and cost variances based on 
contract type. 


9501 99th AREA 

TYPE TYPE PERCENTILE PERCENTILE A-12 ABOVE 

SAMPLE DISTRO VALUE _ VALUE VALUE A-12 

EP> 50% Logistic -47.79 -51.75 -40.11 0.828 

CP >50% Beta -42.73 -54.52 -40.11 0.9372 


Table 18. Contract Type Comparison. 

The data indicate there is a difference in the cost variances based on contract type. 
Compared to fixed-price contracts, 24 percent of programs experienced worse cost 
overruns while only 5 percent of cost-type programs experience worse overruns. 

This apparent difference was investigated using the non-parametric Mann-Whitney 
test. This test determines if there is statistical difference between the mean of two 
populations. On the basis of the data, the test could not reject the null hypothesis that 
there was no difference, with alpha equal to .05. This would refute the apparent 
difference shown in Table 18. Statistically, there is insufficient evidence to claim there is a 
difference in the cost variances experienced by contract type. 

2. Aircraft Programs Versus the Sample 

A second investigation was made to determine if aircraft programs experience 
different overruns than the overall sample. The null hypothesis could not be rejected with 
alpha equal to .05. There is not sufficient evidence to claim there is a difference in aircraft 
program cost variances and the overall sample. 

E. ADJUSTMENTS TO THE CONTRACT BUDGET BASE 

1. The Sample 

All programs examined experienced adjustments to the CBB. In only two of the 
58 contracts examined was there a reduction in the CBB. The mean adjustment to the 
CBB is 53 percent with a median adjustment of 46 percent. 
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The effects of inflation were disregarded based on assumptions stated in Chapter I. 
Hypothesis 4 and 5 were tested using the Mann-Whitney test. For each test alpha was 
selected to be .05. 

1. Hypothesis 4. p cbbfp = p cbbc 

2. Hypothesis 5. p cbbacft = p cbbsamplc 

The alternative hypothesis in each case was the means were not equal. For each 
test, the null hypothesis could not be rejected with rather strong evidence. This indicates 
there is no statistical difference in the adjustments to the Contract Budget Base of 
different types of programs or different types of contracts. It should be noted that while 
the means of the CBB adjustments appear to show a wide difference the tests conducted 
are based on the variances of the data. A summary of the tests and criterion are provided 
in Appendix B. 

2. A-12 CBB Adjustments 

Table 19 shows the EACs developed in Chapter IV for the A-12 program, and the 
percentage change required to adjust the original CBB to the EAC. This percentage 
adjustment is then compared to the distribution of the sample adjustments in CBB. A 
percentile value is determined based on the required adjustment to the A-12 CBB. 

% TYPE VERSUS TYPE VERSUS 

EAC ADJ OISTRO SAMPLE DISTRO SAMPLE 

_ REQD _ W/O ACFT 

7500 57 Logistic 0.53 Normal 0.64 

9069 89.8 Logistic 0.77 Normal 0.88 

11061 131.5 Logistic 0.93 Normal 0.99 

Table 19. Required CBB Adjustments. 

Three EACs are shown in Column 1. The first is the PM’s EAC at termination of 
the Program. The second and third EACs are the Rayleigh derived estimates presented 
earlier. The percentage adjustment that is required to increase the original CBB to the 
estimated EAC is given in Column 2. Column 3 is the distribution best modeling the 
distribution of CBB adjustments for the entire sample. In Column 4 is the percentage of 
programs that required a smaller percentage of adjustment to complete the EMD phase. 
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Columns 5 and 6 a comparison of the A-12 required adjustments versus the sample 
without aircraft included. 

There were not enough data points for fixed-price programs or aircraft programs 
to provide a meaningful comparison. 

3. Analysis 

A CBB adjustment to increase the program funding to $7.5 billion would require a 
57 percent increase in the CBB. A 57 percent increase would result in the program's 
adjustment being greater than about 53 percent of all other programs in the EMD phase. 

A CBB adjustment to increase the program funding to $9,069 billion would have 
required a 90 percent increase in the CBB. This increase would have exceeded 77 percent 
of all other programs. The probability of the A-12 Program costing at least $9 billion is 
.70. 

A CBB adjustment to increase the program funding to $11 billion would have 
required a 132 percent increase in the CBB. This would have exceeded 93 percent of all 
other programs. The probability of the A-12 Program costing at least $11 billion is .49. 

F. COST VARIANCES OF EMD CONTRACTS 

1. Results of the Research 

This research examined fifty-eight total contracts. Table 20 summarizes the 
findings on cost variances in the EMD phase of the acquisition process. 



Table 20. EMD Cost Variances. 

2. Findings 

Fixed-price contracts tend to have the highest cost variance of either contract type. 
The mean cost variance for fixed-price contracts is -21.02 percent. Cost-type contracts 
require the greatest CBB adjustment, 57.24 percent, compared to fixed-price contracts 
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that require a 43 percent increase. Aircraft programs, in general, do not have exceptional 
cost variances, but they do require above average CBB adjustments. 

G. SUMMARY 

This chapter examined the data from the sample 58 contracts and compared the 
results to the A-12 Program. Compared to the sample contracts, the overruns in the A-12 
program, at 41 percent over programmed budget, were exceptional. To increase the 
funding for the program would have required an increase of approximately $5 to $7 
billion. This amount of CBB adjustment would have been exceptional as well. 

The final chapter. Chapter VT, reviews the research questions and the findings. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. REVIEW OF THE RESEARCH QUESTIONS 

1. The Primary Research Question 

Were cost overruns in the A-12 Program exceptional? 

The answer to this question was intended to be a simple yes or no. In keeping 
with this intent the answer is yes. The cost variances in the A-12 program were 
exceptional when compared to the sample contracts in the EMD phase. 

The justification for this finding is based on a comparison of the percentile values 
of the A-12 cost variances, measured against other programs at similar points of 
completion. Table 6, in Chapter IV, shows the cost variances of the A-12. Table 15 in 
Chapter V, provides the percentile values for the A-12 and the area of the distribution that 
exceeded the point values of the cost variances for the A-12. A test to see if the cost 
variances of the A-12 were no different than other aircraft programs was conducted and 
the same results were found. The results of the comparison to other aircraft can be found 
in Table 17, in Chapter V. 

The A-12 cost variances clearly exceeded a majority of other program's cost 
variances. Compared to the sample of 58 contracts, the cost overrun of the A-12 
exceeded 91 percent of the others. At termination, the A-12 exceeded 94 percent of other 
aircraft program's cost overruns. To ensure these results were not biased by the selection 
of the distribution, at least one different distribution was applied in each case to determine 
if like results would be obtained. In each case the cost variance's percentile value 
remained extreme. The comparison distribution's percentile values can be found in 
Appendix B. 

Measurements of only the cost variances do not account for adjustments to the 
Contract Budget Base. A program with a large cost variance can be rebaselined to 
eliminate the variance. Ignoring the CBB adjustments would bias the findings. The 
impact of CBB adjustments was tested in the subsidiary research questions. 
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2. Subsidiary Research Questions 

What would the A-12 FSD program have cost if it continued to completion? 

It is likely that the program would have cost between $9 and $11 billion. Based on 
a Rayleigh model estimate at completion and the MMAE methodology developed by 
Gallagher, the costs and probabilities for the estimates at completion are as follows: 

a. Given a June 1992, first flight date the program would have been 
completed in September 1999. The program would have cost $9 billion. The cumulative 
probability of the program exceeding this cost is .63. 

b. Given a December 1992, first flight date the program would have been 
completed in January 2001. The program would have cost $11 billion. The cumulative 
probability of the program exceeding this cost is .49. 

These results are consistent with the DoD IG estimates contained in Table 3, in 
Chapter III. The DoD IG estimates were prepared by the Cost Analysis Improvement 
Group. 

The budget adjustments that would have been required to finish the program 
would have exceeded between 77 and 93 percent of all other programs. To eliminate the 
cost variance would have required an above average budget adjustment. 

Was the Secretary of Defense justified in terminating the Program for cost 
overruns? 

Yes. The cost variances in the A-12 Program exceeded 97 percent of all other 
programs, as shown in Table 15, in Chapter V. A budget adjustment between 89 and 132 
percent would have been required to finish the program. While this amount of adjustment 
to the budget would be within the distribution of CBB adjustments, (See Table 19) it 
would have as a minimum exceeded 77 percent of all other Contract Budget Base 
adjustments. 

Are there differences in cost overruns between aircraft programs and other 
types of EMD programs? 

No. There is insufficient evidence to conclude that aircraft programs experience 
significantly different cost variances. The test results could not reject the null hypothesis 



that there were no differences. The test showed there was no statistical difference with 
alpha equal to .05. 

Are there differences in cost overruns between fixed-price contracts and cost- 
type contracts for developmental work? 

No. There is not sufficient evidence to claim that fixed-price contracts experience 
statistically different overruns. As measured by the Mann-Whitney non-parametric test, 
the null hypothesis could not be rejected with alpha equal to .05. 

What is the average cost overrun of an EMD contract? 

The mean cost overrun for an EMD contract is 13.37 percent, as shown in Table 
20, in Chapter V. The mean cost overrun for aircraft programs is 14.03 percent. The 
mean cost overrun for fixed-price contracts is 21.02 percent and the mean overrun for 
cost-type contracts is 14.76 percent. The research found that the mean value for overruns 
tends to overstate the extent of the problem. The use of medians provides a more 
accurate picture. The median cost overruns for the above contracts ranged from 6.05 to 
10.2 percent. 

To examine the impact of rebaselining the percentage change in each contract's 
CBB was determined. The results of this examination are provided in Table 7, in Chapter 
IV. To eliminate the possibility that a normal rebaseline would have prevented the 
extraordinary overruns in the A-12 program, the required adjustment to the A-12's CBB 
was examined. 

Depending on the date of first flight (assuming it would be no later than December 
of 1992), the required CBB adjustment is between 89 and 132 percent. An adjustment of 
this magnitude exceeds approximately 77 to 91 percent of all other programs. The 
required mean adjustments were found to vary dependent on program and contract type. 
The results of this investigation are provided in Table 19, in Chapter V. 

There is insufficient evidence to believe that the adjustments to the CBB vary 
depending on either program type or contract type. The apparent difference in mean 
values for the contract types was investigated using the Mann-Whitney test. The test 
showed that the null hypothesis could not be rejected. 
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B. CONCLUSIONS 

1. The Decision to Terminate the Program 

The decision of the Secretary of Defense to terminate the A-12 Program was 
justified The A-12 Program experienced significantly different cost variances from other 
programs examined. Hypothesis 1, the A-12 overruns were no different than other 
programs, was tested to determine if A-12 cost overruns were the same as other EMD 
contracts. This was not proven. At termination the A-12 Program was 41 percent overrun 
from its baseline. This amount of overrun exceeded 97 percent of other programs in the 
sample. The test compared percentile values of the A-12 versus the sample. 

2. Fixed-Price Contracts and Cost Overruns 

The assertion that the cause of the A-12's failure was due to the use of a 
fixed-price contract can not be supported by the data. The selection of contract type is not 
a factor in preventing cost overruns. The fixed-price contracts in the sample had a larger 
overrun than cost-type contracts, 21.02 percent versus 14.76 percent. The median 
overrun values were much closer: 10.2 versus 6.05 percent. Fixed-price contracts required 
smaller budget adjustments than cost contracts. Hypothesis 2 was tested to see if the 
overruns are the same. The null hypothesis, that there was no difference between cost 
overruns in fixed-price contracts and cost contracts, could not be rejected. 

3. Aircraft Program Cost Variances 

The mean cost variance of aircraft programs is the same as other types of 
programs. This indicates that the cost overruns in an aircraft program will be no higher 
than any other type of program. Hypothesis 3, was tested for differences in the mean 
overruns of aircraft programs and other types of programs. The null hypothesis could not 
be rejected, no significant difference exists. The median overruns of the sample examined 
were 6.13 percent for aircraft and 6.09 percent for other types of programs. 

4. Budget Adjustments for Fixed-Price and Cost-Type Contracts 

The required budget adjustments for fixed-price and cost-type contracts are the 
same. The budget adjustment is the amount, measured by percentage, required to adjust 
the initially specified budget to the ending budget. In other words, it is the amount of 
additional money the program manager needs to finish the program. The budget 
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adjustment required to finish the A-12 Program would have been between a 77 and 93 
percent increase. The mean budget adjustment for fixed-price contracts is 44 percent and 
the mean budget adjustment for cost-type contracts is 57 percent. Hypothesis 4 was 
tested to see if there are differences between the Contract Budget Base adjustments for 
fixed-price contracts and the CBB adjustments for cost-type contracts. 

A visual inspection of the data would indicate a difference, but the Mann-Whitney 
test showed no statistical difference. This can be accounted for by the dispersion of the 
data. Prior to conducting the test the data was normalized by dividing each data point by 
its standard deviation. The null hypothesis, that there are no differences, could not be 
rejected. 

5. Budget Adjustments for Aircraft Programs 

Aircraft programs require about the same amount of budget adjustment as other 
types of programs. The mean budget adjustment for aircraft programs was higher than 
other types of programs. The dispersion of the values was much greater for aircraft 
programs than for other types of programs, and the data were normalized prior to testing. 
Hypothesis 5 was tested to determine if aircraft programs experience greater mean CBB 
adjustments than other programs. The null hypothesis could not be rejected. There is no 
statistical difference in the mean budget adjustments of aircraft programs and the sample. 

6. Mean Values Versus Median Values 

Mean values overstate cost overruns. To compare a program's overruns with 
other programs, the median cost overrun should be used. The mean is a good measure of 
central tendency for data modeled by the Normal distribution. Cost overruns are modeled 
by the Beta distribution. The use of the median is a more accurate measure of central 
tendency for skewed data. As shown in Figure 3, in Chapter II, the distribution of cost 
overruns is skewed to the left. The use of mean values can be misleading. For example, 
the mean cost variance of fixed-price contracts and cost-type contracts is -21.02 percent 
and -14.76 percent respectively. At first glance the conclusion could be drawn that 
fixed-price contracts are significantly different from cost-type contracts. 

The median values for the cost variances of fixed-price and cost-type contracts are 
-10.2 percent and -6.05 percent respectively. The median values show less difference by 
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contract type. The test of the hypothesis found no statistical difference in cost variances 
based on contract type. 

7. Cost Variances of EMD Contracts 

The results of this research agree with prior research, cost overruns do not get 
better with time. Programs less than 10 percent complete have a positive cost variance 
(cost underrun). From 11 percent complete on programs tend to experience cost 
overruns. From 11 to 80 percent complete the cost overrun increases with program 
length. The largest cost overrun for the sample, was from 71 to 80 percent complete and 
was negative 14.52 percent. The mean value of the cost variance of the sample was 
negative 13.37 percent while the median value was negative 6.09 percent Very few 
contracts were found to have positive cost variances at any point in the program. A 
summary of the findings on cost variances is provided in Table 11, Chapter IV, and Table 
20 in Chapter V. 

The observations of Abba and Christie and the empirical findings of Heise were 
confirmed in that the cost variances tended to worsen as the program progressed. This is 
shown in Table 11, in Chapter IV. 

8. Causes of Cost Growth 

The cause of the A-12's cost overruns are difficult to determine. It was not the 
intent of this thesis to provide the answer to why the A-12 had exceptional overruns. 
Based on the data examined, it appears that the Program encountered significantly greater 
technical challenges than were anticipated, which delayed first flight and increased cost. 

The contracts examined in this research all showed variances from the baseline. 
The majority of these contracts experienced cost overruns throughout the program. The 
Rand report [Ref. 10] found no single cause for the cost growth of weapon system 
programs. The report found a correlation between the size of the program and the length 
of the program and cost growth. The smaller the program, the greater the variance and 
the longer the program is carried out, the greater the variance. 

A 1992 GAO report summarizes the problem of determining the cause of cost 
growth: 
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The weapon system problems we have reported in the past 2 years 
mirror those we reported in the 1970s and the 1980s. This does not imply 
that the history of weapon acquisitions amounts to a string of bad 
programs. The point is that, despite conscious attempts to improve the 
acquisition process, weapons still cost more, take longer to field, often 
encounter performance problems, and, in many instances are difficult to 
produce or support. The persistence of these problems reflects the fact 
that the design, development, and production of major weapon systems are 
extremely complex technical processes. . .In short, it takes a myriad of 
things to go right for a program to be successful, but only a few things to 
go wrong to cause major problems. [Ref. 27: p. 15] 

C. RECOMMENDATIONS 

1. The A-12 

The Government's decision to terminate the fixed-price contract of the A-12 is 
justified, based on an examination of the data used in this research. The Government 
should continue to pursue the case in court and demand reimbursement for the 
unliquidated progress payments. 

2. Cost Estimating 

The findings of this research indicate that estimates at completion in EMD 
contracts are too optimistic. As shown in Table 7, the mean adjustment to the CBB is 54 
percent. Only two of the 58 contracts showed downward revisions in the CBB. More 
emphasis should be placed on developing realistic estimates of the cost of the contract. A 
cost underrun in the early stages of a program is a poor indicator of future cost 
performance. Fully defining the scope of work before the start of EMD may help alleviate 
the problem. A longer Demonstration and Validation phase may be warranted in 
programs with substantial technical risk. 

3. Fixed-Price Contracts for EMD 

The belief that fixed-price contracts place the majority of the cost risk on the 
contractor is not supported by the data for EMD contracts. No statistical difference was 
shown in the median values for cost-type contracts and fixed-price contracts. While 
fixed-price contracts showed more extreme points than cost-type contracts, this may have 
been due to the Government's reluctance to adjust the price of the fixed-price type 
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contracts. As shown in Table 10 the Air Force has used a fixed-price type contract in the 
majority of EMD contracts that were examined. 

The decision to use a cost-type contract will not have any impact on the cost 
variance of the contract. The mean budget adjustment for fixed-price contracts was 44 
percent, the median value was 54 percent, while the mean budget adjustment for cost-type 
contracts was 57 percent and the median value was 45 percent. By examining the means 
for budget adjustments the conclusion could be drawn that the use of a fixed-price 
contract would require a smaller budget adjustment. The use of median values would 
show that the use of a fixed-price contract would require a greater budget adjustment. 

The mean values can be skewed by outliers, extreme values by very few programs. 

D. AREAS FOR FURTHER RESEARCH 

1. The Rayleigh Distribution Model 

The Rayleigh model shows promise as a cost estimating tool. Further research 
into optimizing this model for more narrowly defined programs should be examined. 
Adjustment of the parameters of the model resulted in greater accuracy. This research 
used only ten values to optimize the parameters for aircraft contracts. By using more 
values to optimize the parameters an even more refined model could be achieved 

The Rayleigh model proved to be a better estimator of costs at completion than the 
CPI techniques given an unstable baseline. In programs where the baseline is relatively 
stable the CPI technique is both easy to use and accurate. The advantage that the 
Rayleigh technique provides is the ability to provide estimates independent of the baseline. 

It was not the intent of this research to prove the value of the Rayleigh model. 

The model was selected based on its ability to provide EACs independent of current 
estimates of final costs. Other techniques exist for estimating costs that were also not 
examined. 

2. Fixed-Price Contracts 

This research indicates that the use of fixed-price contracts in developmental work 
is not inherently bad. All contracts examined had adjustments made to the CBB regardless 
of contract type. Additional work could examine in more detail the differences in cost 
variances between cost-type contracts and fixed-price type contracts. 
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APPENDIX A. CONTRACT DATABASE 


This appendix contains the contracts used in the research. The agreement with 
OUSD (A&T) requires that the name of the contract and the contractor not be disclosed. 
The contracts are listed by number. The corresponding list of contracts can be released 
with the permission of the OUSD (A&T). The list of contracts used in this research is 
maintained by Dr. David V. Lamm, Systems Management Department, Naval 
Postgraduate School. 

ABBREVIATIONS USED 

1 ACWP. The reported actual cost of work performed drawn from the database. 13. 

2. ALLOC BUDGET. Actual budget committed to the program. 

3. BCWP The reported budgeted cost of work performed drawn from the database. 

4. CBB. Contract Budget Base. 

5. COMPLPT. Percentage completion point. As measured by BCWP/ALLOC 
BUDGET. 

6. CP. Cost Plus. 

7. CPAF. Cost-Plus-Award-Fee. 

8 CPIF. Cost-Plus-Incentive-Fee. 

9. CV. Cost Variance. Percentage cost variance determined by (BCWP-ACWP)/BCWP. 

10. FP Firm-Fixed-Price 

11. FPIF. Fixed-Price-Incentive-Firm. 

12. FSD. Full-Scale Development. Equivalent to Engineering and Manufacturing 
Development (EMD). Drawn from the description in the DAES database. "DEV" as 
used in the DAES database is equivalent to EMD. 

13. K. Contract. 

14. K TYPE. Contract type. 

15. SYSTEM The type of system the contract was for. General categories were 
established. 
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SERVICE: 

NAVY 


SYSTEM: 

AIRCRAFT 


PHASE: 
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APPENDIX B. STATISTICAL INFORMATION 


This appendix provides the detailed statistical information to support information 
contained in the text. The appendix is subdivided into the following sections: 

A. Modeling of the Data. 

B. Distribution Information 
C Beta Value Conversions 
D. Non-parametric Tests 

A. MODELING OF THE DATA 

The accuracy of the A-12 percentile values is dependent on how well the data were 
modeled by the specified parameters. To ensure that the percentile values were not biased 
by the choice of distribution, each data set was checked with a second distribution. This 
section contains the graphical representations of the distribution of the data. The graphs 
in this section were produced using BestFit [Ref. 20], 

1. Sample: 0 to 10 Percent Complete 


Figure 7 shows the distribution of cost variances for all contracts from 0 to 10 
percent complete. The data were best modeled by the Normal distribution, with mean 
equal to 1.79 and standard deviation of 11.56. 
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2. Sample Data: 11 to 20 Percent Complete. 

Figure 8 shows the distribution for cost variances of the sample, from 11 -20 
percent complete. Thirty-seven contracts were included in the sample. The data were 
skewed to the left, and were best represented by the Beta distribution with parameters 
(2.3, 1,07). 



Figure 8. Sample Data 11 to 20 Percent Complete. 

3. Sample Data: 21 to 30 Percent Complete 

Figure 9 is the distribution of cost variances for contracts 21 to 30 percent 
complete. Thirty-four contracts constitute the sample. The data were centered near zero 
and were modeled by the Logistic distribution with parameters (-5.27, 11.10). 



Figure 9. Sample Data 21 to 30 Percent Complete. 
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4. Sample Data 31 to 40 Percent Complete 

Figure 10 is the distribution of cost variances in programs 31 to 40 percent 
complete. Thirty-seven contracts constitute the sample. The sample is modeled by the 
Logistic distribution with parameters (-8.08, 5.54). 


Comparison of Input Distribution and Logistic(-8.08,5.54) 
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Figure 10. Sample Data 31 to 40 Percent Complete. 

5. Sample Data 41 to 50 Percent Complete 

Figure 11 is the distribution of cost variances in programs 41 to 50 percent 
complete. The data are skewed to the left. However, the degree of skewness does not 
warrant the selection of a Beta distribution. The Logistic distribution models the data. 
The parameters of the distribution are (-11.02, 8.08). 



Figure 11. Sample Data 41 to 50 Percent Complete. 
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6. Aircraft Programs: 0 to 10 Percent Complete 


Figure 12 provides the distribution of cost variances within aircraft programs, from 
0 to 10 percent complete. Eleven contracts are used. The distribution is centered near 
zero. The data are modeled by the Beta distribution with parameters (1.75, .83). 
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Figure 12. Aircraft Programs 0 to 10 Percent Complete. 


7. Aircraft Programs: 11 to 20 Percent Complete 


Figure 13 provides the distribution of cost variances in aircraft programs 11 to 20 
percent complete. The Beta distribution modeled the data. Eleven contracts were in the 
sample. The parameters of the distribution are (1.17, .56). 
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Figure 13. Aircraft Programs 11 to 20 Percent Complete. 
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8. Aircraft Programs: 21 to 30 Percent Complete 

Figure 14 provides the cost variance distribution for aircraft programs 21 to 30 
percent complete. The Beta distribution models the data with parameters (1.43, .64). 
Twelve contracts were included in the sample. 
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Figure 14. Aircraft Programs 21 to 30 Percent Complete. 

9 . Aircraft Programs: 31 to 40 Percent Complete 
Figure 15 provides the distribution of cost variances in aircraft programs 31 to 40 


percent complete. The data are skewed to the left and are modeled by the Beta 
distribution. Fourteen contracts were included in the sample. The parameters of the 
distribution are (2.05, .94). 
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10. Aircraft Programs: 41 to 50 Percent Complete 

Figure 16 provides the distribution of cost variances within aircraft programs 41 
to 50 percent complete. The data are modeled by the Logistic distribution with 
parameters (-12.63, 10.16). Ten contracts were included in the sample. 

Comparison of Input Distribution and Logistical2.63,10.16) 
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Figure 16. Aircraft Programs 41 to 50 Percent Complete. 

11. Sample Contracts Cost Variance Distribution 

Figure 17 provides the distribution of cost variances for all programs that had met 
or exceeded 80 percent complete. Forty contracts were included in the sample. The 
distribution was skewed to the left and was modeled by the Beta distribution. The 
parameters of the distribution are (1.49, .82). 
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Figure 17. All Contracts at Least 80 Percent Complete. 
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12. Distribution of Cost Contracts 

Figure 18 provides the distribution of cost variances, for programs with cost-type 
contracts, at least 50 percent complete. The data are skewed to the left and are modeled 
by the Beta distribution. Thirty-nine contracts were included in the sample. The 
parameters of the distribution are (1.65, .40). 


Comparison of Input Distribution and Beta(1.65,0.40) * 64.01 
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Figure 18. All Cost-Type Contracts at Least 50 Percent Complete. 

13. Distribution of Fixed-Price Contracts 

Figure 19 provides the cost variances for programs with fixed-price type contracts 
that are at least 50 percent complete. The number of contracts included in the sample is 
15. The parameters of the Beta distribution are ( .95, .22). 



Figure 19. All Fixed-Price Contracts at Least 50 Percent Complete. 
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IS. Distribution of Contract Budget Base Adjustments 

Figure 20 shows the distribution of adjustments to the Contract Budget Base, for 
all programs. The programs included had to be at least 70 percent complete and have 50 
percent of the total program included in the cost reports. The distribution of CBB 
adjustments is modeled by the Logistic distribution with parameters (53.57, 30.21). 
Twenty-nine contracts were included in the sample. 



Figure 20. Contract Budget Base Adjustments for the Sample. 


B. DISTRIBUTION INFORMATION 


1. Statistics of the Distributions 

Table 21 gives the detailed statistics of the distributions that best modeled the data. 
Column 1 is the type sample that is modeled. Column 2 provides the percentage 
completion brackets. In Columns 3 and 4 are the distribution's parameters. In Columns 5 
and 6 are the extreme values for the data. The extreme values do not include outliers that 
were discarded. The best distribution that modeled the data and the second best 
distribution are given in Columns 7 and 8. The accompanying Chi-squared values are 
provided in Columns 9 and 10. 


96 
















2. Percentile Value Information 

Table 22 contains the percentile values for the samples based on the best and 
second best distributions. Columns 1 and 2 give the sample type and the percentage 
completion bracket. Columns 3 and 4 provide the 0 to 1 Beta value, if applicable. In 
Columns 5 and 6 are the .01 percentile and .05 percentile values for the given distribution. 
The next best percentile values are given in Columns 7 and 8. 
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C. BETA VALUE CONVERSIONS 

The Beta distribution is a continuous distribution, with range 0 to 1. Cost 
variances can vary from negative infinity to positive infinity. To model cost variances 
using a Beta distribution the observed values must be converted to a range of 0 to 1. This 
is accomplished by using the following formula: Z>(P: v, w) + a[ 1 -(p: v, w)]. Where 
o>x>b and (P: v,vF) = -u)^'du. [Ref. 12] 

The observed cost variances tended to assume one of two shapes. If the 
distribution was skewed to the left, with a majority of the observations clustering close to 
zero, then the distribution assumed a J shape, characteristic of the Beta distribution with 
(v-l)(w-l) < 0. If a relatively large number of observations occurred in the tail of the 
distribution then the distribution tended to assume a U shape where v < 1 and w < 1. 

The estimated parameters provided by BestFit were used to obtain the percentile 
values The percentile values were then converted to values ranging from a < x < b by 
arithmetic manipulation of the original formula. 

The distributions shown in Section A, have a Beta distribution ranging from 0 to 1 
superimposed over data ranging from a to b. 

D. NON-PARAMETRIC TESTS OF THE HYPOTHESES 

1. The Mann-Whitney Test 

The Mann-Whitney Test tests for differences between the means of two 
populations. The test statistic is given by: U- py/ou = Z. For a two tailed test the 
decision criterion is: Reject H„ if Z > -z^ or Z < z^. For all four tests alpha is set at .05. 
Therefore, a/2 equals .025. The value of z a/2 equals 1.96. 

2. Hypothesis 2 

(ifp = p c . This hypothesis tests for differences in the means of fixed-price contracts 
and the mean of cost-type contracts. H,,, the null hypothesis, is the mean cost variance of 
fixed-price contracts equals the mean of cost-type contracts. H A , the alternative 
hypothesis, is the means are not equal. This is a two tailed test. The purpose of this test 
is to determine if the choice of contract type contributed to the exceptional cost overruns 
in the A-12 program. 

A requirement of the Mann-Whitney test is the populations must be the same, 
except for possible locations of the mean. The standard deviations of cost-type contracts 
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and fixed-price type contracts was significantly different: 28.23 and 16.13 respectively. 

To standardize the two populations, each set of data was divided by its standard deviation 
before conducting the test. The test statistics for Hypothesis 2 are as follows: 

From the results of the test the null hypothesis can not be rejected. There is no 
statistical difference in the means of the two populations. The test statistics for this test 
are given below: 

Test 

R1 _R2_nl_n2_ Statistic 

381 1104 15 39 0.6083 

3. Hypothesis 3 

l^acft = sample Hypothesis 3 tests for differences in the cost variances of aircraft 
contracts and other programs. This test examines if aircraft programs experience 
statistically different cost variances than other type of programs. The mean cost variance 
of aircraft programs is -14.03 with standard deviation 15.69. The mean cost variance of 
other programs in the sample is -14.10 with standard deviation 16.43. No difference was 
assumed between the two populations. As would be expected, with similar means and 
standard deviations, the null hypothesis could not be rejected. The test statistics are given 
below: 

Test 

R1 _R2_nl_n2_ Statistic 

324 711 15 30 0.5056 

4. Hypothesis 4 

|T C bbfp = M-ctbc- Hypothesis 4 is similar to Hypothesis 2. Hypothesis 2 tested for 
differences in the cost variances between fixed-price type and cost-type contracts. 
Hypothesis 4 tests for differences in the mean budget adjustments. The mean CBB 
adjustment for fixed-price type contracts is 43.93 with standard deviation 59.84. The 
mean CBB adjustment for cost-type contracts is 57.24 with standard deviation 54.37. 
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The test statistics, for Hypothesis 4, are given below: 


Test 

R1 _R2_nl_n2_ Statistic 

115 320 8 21 0.244 

The null hypothesis could not be rejected. There is no statistical difference 
between the mean CBB adjustments based on contract type. 

5. Hypothesis 5. 

f^cbbacft = M'cbbsampit- Hypothesis 5 tests for differences in the CBB adjustments in 
aircraft programs versus other types of programs. The mean CBB adjustment for aircraft 
programs is 94.15 with standard deviation 86.15. The mean CBB adjustment for other 
types of programs in the sample is 42.99 with standard deviation 40.20. The test statistics 
are given below: 

Test 

R1 _R2_nl_n2_ Statistic 

80 355 6 23 0.5384 

i____ 

The two populations were divided by their respective standard deviations before 
the test to eliminate the difference in variance between the two populations. The test 
results indicate that the null hypothesis can not be rejected. Despite the apparent 
differences in the means of the two populations, there is insufficient evidence to claim that 
aircraft programs mean budget adjustments are greater than other programs. 
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